2024-01-16T04:37:21.843 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-16T04:37:21.843 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi --owner scheduled_yuriw@teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517339 --description rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} -- /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517339/orig.config.yaml 2024-01-16T04:37:21.875 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T04:37:21.957 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517339 branch: wip-yuri5-testing-2024-01-11-1300-pacific description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} email: yweinste@redhat.com first_in_suite: false job_id: '7517339' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri5-testing-2024-01-11-1300-pacific ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: vault rgw crypt vault auth: token rgw crypt vault prefix: /v1/transit/ rgw crypt vault secret engine: transit setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(POOL_APP_NOT_ENABLED\) sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: default sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f rgw: client.0: use-vault-role: client.0 workunit: branch: wip-yuri5-testing-2024-01-11-1300-pacific sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9013 sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri5-testing-2024-01-11-1300-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f targets: smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNs8qpHHPPUS2s7++BnIXXAgH3hvybrKv4GM3m7Zbb48nI7wNNZl355X3XAuPc/Bw79V8QAbjDfp0cJPFstEVA4= tasks: - install: null - ceph: null - vault: client.0: engine: transit install_sha256: 7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458 install_url: https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip prefix: /v1/transit/keys/ root_token: test_root_token secrets: - path: my-key-1 - path: my-key-2 - rgw: client.0: null - s3tests: client.0: barbican: kms_key: my-key-1 kms_key2: my-key-2 force-branch: ceph-pacific vault_kv: key_path: my-key-1 key_path2: my-key-2 vault_old: key_path: my-key-1/1 key_path2: my-key-2/1 vault_transit: key_path: my-key-1 key_path2: my-key-2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-15_22:05:22 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-16T04:37:21.957 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa; will attempt to use it 2024-01-16T04:37:21.958 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks 2024-01-16T04:37:21.958 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-16T04:37:21.960 INFO:teuthology.task.internal:Checking packages... 2024-01-16T04:37:21.988 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a70aebef70df6ec2d957347ccbfc484e0573dd5f' 2024-01-16T04:37:21.988 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-16T04:37:21.989 INFO:teuthology.packaging:ref: None 2024-01-16T04:37:21.989 INFO:teuthology.packaging:tag: None 2024-01-16T04:37:21.989 INFO:teuthology.packaging:branch: wip-yuri5-testing-2024-01-11-1300-pacific 2024-01-16T04:37:21.989 INFO:teuthology.packaging:sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f 2024-01-16T04:37:21.989 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri5-testing-2024-01-11-1300-pacific 2024-01-16T04:37:22.210 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-449.ga70aebef 2024-01-16T04:37:22.212 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-16T04:37:22.218 INFO:teuthology.task.internal:no buildpackages task found 2024-01-16T04:37:22.218 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-16T04:37:22.228 INFO:teuthology.task.internal:Saving configuration 2024-01-16T04:37:22.239 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-16T04:37:22.245 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T04:37:22.272 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517339', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 04:32:17.387869', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNs8qpHHPPUS2s7++BnIXXAgH3hvybrKv4GM3m7Zbb48nI7wNNZl355X3XAuPc/Bw79V8QAbjDfp0cJPFstEVA4='} 2024-01-16T04:37:22.272 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-16T04:37:22.278 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-01-16T04:37:22.278 INFO:teuthology.run_tasks:Running task console_log... 2024-01-16T04:37:22.320 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f18d66d0ee0>, signals=[15]) 2024-01-16T04:37:22.321 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-16T04:37:22.327 INFO:teuthology.task.internal:Opening connections... 2024-01-16T04:37:22.328 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2024-01-16T04:37:22.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T04:37:22.405 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-16T04:37:22.412 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2024-01-16T04:37:22.433 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2024-01-16T04:37:22.434 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2024-01-16T04:37:22.494 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="CentOS Stream" 2024-01-16T04:37:22.494 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="8" 2024-01-16T04:37:22.494 INFO:teuthology.orchestra.run.smithi156.stdout:ID="centos" 2024-01-16T04:37:22.494 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="rhel fedora" 2024-01-16T04:37:22.494 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="8" 2024-01-16T04:37:22.495 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el8" 2024-01-16T04:37:22.495 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-16T04:37:22.495 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2024-01-16T04:37:22.495 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-16T04:37:22.495 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://centos.org/" 2024-01-16T04:37:22.495 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-16T04:37:22.495 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-16T04:37:22.495 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-16T04:37:22.496 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2024-01-16T04:37:22.511 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-16T04:37:22.519 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-16T04:37:22.524 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-16T04:37:22.524 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2024-01-16T04:37:22.553 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-16T04:37:22.559 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-16T04:37:22.559 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2024-01-16T04:37:22.623 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-16T04:37:22.657 INFO:teuthology.run_tasks:Running task kernel... 2024-01-16T04:37:22.671 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-16T04:37:22.671 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-16T04:37:22.671 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-16T04:37:22.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T04:37:22.691 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2024-01-16T04:37:22.750 INFO:teuthology.orchestra.run.smithi156.stdout:4.18.0-489.el8.x86_64 2024-01-16T04:37:22.750 INFO:teuthology.task.kernel:Running kernel on smithi156: 4.18.0-489.el8.x86_64 2024-01-16T04:37:22.751 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum install -y kernel 2024-01-16T04:37:27.713 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - AppStream 15 MB/s | 35 MB 00:02 2024-01-16T04:37:36.329 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - BaseOS 28 MB/s | 58 MB 00:02 2024-01-16T04:37:44.854 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - Extras 66 kB/s | 18 kB 00:00 2024-01-16T04:37:45.412 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2024-01-16T04:37:45.706 INFO:teuthology.orchestra.run.smithi156.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-01-16T04:37:46.682 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 20 MB/s | 16 MB 00:00 2024-01-16T04:37:50.523 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 392 kB/s | 24 kB 00:00 2024-01-16T04:37:54.290 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-16T04:37:54.290 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-16T04:37:54.585 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:37:54.585 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:37:54.585 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2024-01-16T04:37:54.585 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:37:54.586 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:37:54.586 INFO:teuthology.orchestra.run.smithi156.stdout: kernel x86_64 4.18.0-532.el8 baseos 10 M 2024-01-16T04:37:54.586 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-01-16T04:37:54.586 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-core x86_64 4.18.0-532.el8 baseos 43 M 2024-01-16T04:37:54.586 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules x86_64 4.18.0-532.el8 baseos 36 M 2024-01-16T04:37:54.586 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:37:54.586 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:37:54.586 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:37:54.586 INFO:teuthology.orchestra.run.smithi156.stdout:Install 3 Packages 2024-01-16T04:37:54.586 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:37:54.587 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 90 M 2024-01-16T04:37:54.587 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 96 M 2024-01-16T04:37:54.587 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:37:55.358 INFO:teuthology.orchestra.run.smithi156.stdout:(1/3): kernel-4.18.0-532.el8.x86_64.rpm 18 MB/s | 10 MB 00:00 2024-01-16T04:37:56.442 INFO:teuthology.orchestra.run.smithi156.stdout:(2/3): kernel-modules-4.18.0-532.el8.x86_64.rpm 22 MB/s | 36 MB 00:01 2024-01-16T04:37:56.892 INFO:teuthology.orchestra.run.smithi156.stdout:(3/3): kernel-core-4.18.0-532.el8.x86_64.rpm 21 MB/s | 43 MB 00:02 2024-01-16T04:37:56.892 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:37:56.892 INFO:teuthology.orchestra.run.smithi156.stdout:Total 39 MB/s | 90 MB 00:02 2024-01-16T04:37:57.573 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:37:58.000 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:37:58.001 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:37:59.306 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:37:59.334 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:38:02.128 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:38:02.681 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-core-4.18.0-532.el8.x86_64 1/3 2024-01-16T04:38:06.276 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-core-4.18.0-532.el8.x86_64 1/3 2024-01-16T04:38:06.655 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-modules-4.18.0-532.el8.x86_64 2/3 2024-01-16T04:38:10.079 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-modules-4.18.0-532.el8.x86_64 2/3 2024-01-16T04:38:10.378 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-4.18.0-532.el8.x86_64 3/3 2024-01-16T04:38:32.236 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-core-4.18.0-532.el8.x86_64 3/3 2024-01-16T04:38:32.260 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-modules-4.18.0-532.el8.x86_64 3/3 2024-01-16T04:38:32.594 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-4.18.0-532.el8.x86_64 3/3 2024-01-16T04:38:32.594 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-4.18.0-532.el8.x86_64 1/3 2024-01-16T04:38:32.594 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-core-4.18.0-532.el8.x86_64 2/3 2024-01-16T04:38:33.413 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-modules-4.18.0-532.el8.x86_64 3/3 2024-01-16T04:38:33.413 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:38:33.413 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:38:33.413 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-4.18.0-532.el8.x86_64 kernel-core-4.18.0-532.el8.x86_64 2024-01-16T04:38:33.413 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules-4.18.0-532.el8.x86_64 2024-01-16T04:38:33.413 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:38:33.413 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:38:33.549 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q kernel | sort -rV | head -n 1 2024-01-16T04:38:33.786 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-4.18.0-532.el8.x86_64 2024-01-16T04:38:33.787 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-532.el8.x86_64 2024-01-16T04:38:33.787 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-532.el8.x86_64 2024-01-16T04:38:33.787 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-16T04:38:33.787 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-16T04:38:33.787 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-01-16T04:38:33.787 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi156.front.sepia.ceph.com, path=None, version=distro) 2024-01-16T04:38:33.787 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q kernel | sort -rV | head -n 1 2024-01-16T04:38:33.998 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-4.18.0-532.el8.x86_64 2024-01-16T04:38:33.999 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-532.el8.x86_64 2024-01-16T04:38:33.999 DEBUG:teuthology.orchestra.run.smithi156:> sudo rpm -qi grub2-tools 2024-01-16T04:38:34.032 INFO:teuthology.orchestra.run.smithi156.stdout:Name : grub2-tools 2024-01-16T04:38:34.032 INFO:teuthology.orchestra.run.smithi156.stdout:Epoch : 1 2024-01-16T04:38:34.033 INFO:teuthology.orchestra.run.smithi156.stdout:Version : 2.02 2024-01-16T04:38:34.033 INFO:teuthology.orchestra.run.smithi156.stdout:Release : 129.el8 2024-01-16T04:38:34.033 INFO:teuthology.orchestra.run.smithi156.stdout:Architecture: x86_64 2024-01-16T04:38:34.033 INFO:teuthology.orchestra.run.smithi156.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-16T04:38:34.033 INFO:teuthology.orchestra.run.smithi156.stdout:Group : System Environment/Base 2024-01-16T04:38:34.033 INFO:teuthology.orchestra.run.smithi156.stdout:Size : 9265586 2024-01-16T04:38:34.033 INFO:teuthology.orchestra.run.smithi156.stdout:License : GPLv3+ 2024-01-16T04:38:34.033 INFO:teuthology.orchestra.run.smithi156.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-16T04:38:34.033 INFO:teuthology.orchestra.run.smithi156.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-16T04:38:34.033 INFO:teuthology.orchestra.run.smithi156.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-16T04:38:34.033 INFO:teuthology.orchestra.run.smithi156.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-16T04:38:34.034 INFO:teuthology.orchestra.run.smithi156.stdout:Relocations : (not relocatable) 2024-01-16T04:38:34.034 INFO:teuthology.orchestra.run.smithi156.stdout:Packager : CentOS BuildSystem 2024-01-16T04:38:34.034 INFO:teuthology.orchestra.run.smithi156.stdout:Vendor : CentOS 2024-01-16T04:38:34.034 INFO:teuthology.orchestra.run.smithi156.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-16T04:38:34.034 INFO:teuthology.orchestra.run.smithi156.stdout:Summary : Support tools for GRUB. 2024-01-16T04:38:34.034 INFO:teuthology.orchestra.run.smithi156.stdout:Description : 2024-01-16T04:38:34.034 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:38:34.034 INFO:teuthology.orchestra.run.smithi156.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-16T04:38:34.034 INFO:teuthology.orchestra.run.smithi156.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-16T04:38:34.034 INFO:teuthology.orchestra.run.smithi156.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-16T04:38:34.034 INFO:teuthology.orchestra.run.smithi156.stdout:hardware devices. 2024-01-16T04:38:34.034 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:38:34.034 INFO:teuthology.orchestra.run.smithi156.stdout:This subpackage provides tools for support of all platforms. 2024-01-16T04:38:34.035 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-16T04:38:34.035 INFO:teuthology.task.kernel:Updating grub on smithi156 to boot 4.18.0-532.el8.x86_64 2024-01-16T04:38:34.035 DEBUG:teuthology.orchestra.run.smithi156:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-16T04:38:35.681 INFO:teuthology.orchestra.run.smithi156.stderr:Generating grub configuration file ... 2024-01-16T04:38:37.783 INFO:teuthology.orchestra.run.smithi156.stderr:done 2024-01-16T04:38:37.785 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2024-01-16T04:38:37.811 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.kdhfUrW9Nd 2024-01-16T04:38:37.811 DEBUG:teuthology.orchestra.run.smithi156:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kdhfUrW9Nd 2024-01-16T04:38:37.895 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.kdhfUrW9Nd 2024-01-16T04:38:38.029 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.kdhfUrW9Nd is 6KB 2024-01-16T04:38:38.077 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.kdhfUrW9Nd 2024-01-16T04:38:38.094 DEBUG:teuthology.orchestra.run.smithi156:> sudo /bin/ls /boot/loader/entries || true 2024-01-16T04:38:38.170 INFO:teuthology.orchestra.run.smithi156.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-16T04:38:38.171 INFO:teuthology.orchestra.run.smithi156.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-16T04:38:38.171 INFO:teuthology.orchestra.run.smithi156.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-16T04:38:38.171 INFO:teuthology.orchestra.run.smithi156.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-532.el8.x86_64.conf 2024-01-16T04:38:38.172 DEBUG:teuthology.orchestra.run.smithi156:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-532.el8.x86_64 2024-01-16T04:38:38.321 DEBUG:teuthology.orchestra.run.smithi156:> sudo shutdown -r now 2024-01-16T04:39:08.352 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-16T04:39:08.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-01-16T04:39:08.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T04:39:26.784 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2024-01-16T04:39:35.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-01-16T04:39:35.793 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T04:39:38.844 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2024-01-16T04:39:50.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-01-16T04:39:50.857 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T04:40:09.276 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2024-01-16T04:40:24.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-01-16T04:40:24.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T04:40:24.652 DEBUG:teuthology.orchestra.run.smithi156:> true 2024-01-16T04:40:24.958 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-01-16T04:40:24.958 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-532.el8.x86_64"... 2024-01-16T04:40:24.958 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2024-01-16T04:40:24.975 INFO:teuthology.orchestra.run.smithi156.stdout:4.18.0-532.el8.x86_64 2024-01-16T04:40:24.975 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-532.el8.x86_64 vs 4.18.0-532.el8.x86_64 2024-01-16T04:40:24.975 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-16T04:40:24.975 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-16T04:40:25.977 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-16T04:40:25.977 DEBUG:teuthology.orchestra.run.smithi156:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-16T04:40:26.725 INFO:teuthology.orchestra.run.smithi156.stdout:ttyS1 2024-01-16T04:40:26.750 DEBUG:teuthology.parallel:result is None 2024-01-16T04:40:26.751 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-16T04:40:26.758 INFO:teuthology.task.internal:Creating test directory... 2024-01-16T04:40:26.758 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-16T04:40:27.010 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-16T04:40:27.017 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-16T04:40:27.063 INFO:teuthology.task.internal:Creating archive directory... 2024-01-16T04:40:27.063 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-16T04:40:27.095 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-16T04:40:27.101 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-16T04:40:27.101 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-16T04:40:27.281 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T04:40:27.292 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T04:40:27.294 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-16T04:40:27.301 INFO:teuthology.task.internal:Configuring sudo... 2024-01-16T04:40:27.301 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-16T04:40:27.329 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-16T04:40:27.422 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-16T04:40:27.422 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-16T04:40:27.463 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T04:40:27.547 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T04:40:27.608 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T04:40:27.652 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T04:40:27.695 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:40:27.695 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-16T04:40:27.762 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2024-01-16T04:40:27.858 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T04:40:27.875 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-16T04:40:27.883 INFO:teuthology.task.internal:Starting timer... 2024-01-16T04:40:27.884 INFO:teuthology.run_tasks:Running task pcp... 2024-01-16T04:40:27.915 INFO:teuthology.run_tasks:Running task selinux... 2024-01-16T04:40:27.924 DEBUG:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2024-01-16T04:40:28.028 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: 2024-01-16T04:40:28.031 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-16T04:40:28.031 DEBUG:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2024-01-16T04:40:28.073 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2024-01-16T04:40:28.074 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi156.front.sepia.ceph.com': 'permissive'} 2024-01-16T04:40:28.074 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T04:40:28.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T04:40:28.143 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-16T04:40:28.143 DEBUG:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2024-01-16T04:40:28.210 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-16T04:40:28.220 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-16T04:40:28.226 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-16T04:40:28.398 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-16T04:40:28.413 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-16T04:40:28.415 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi156.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-16T04:46:17.265 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi156.front.sepia.ceph.com')] 2024-01-16T04:46:17.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-01-16T04:46:17.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T04:46:17.342 DEBUG:teuthology.orchestra.run.smithi156:> true 2024-01-16T04:46:17.410 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-01-16T04:46:17.410 INFO:teuthology.run_tasks:Running task clock... 2024-01-16T04:46:17.420 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-16T04:46:17.421 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-16T04:46:17.421 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T04:46:17.481 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-16T04:46:17.495 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-16T04:46:17.551 INFO:teuthology.orchestra.run.smithi156.stderr:sudo: ntpd: command not found 2024-01-16T04:46:17.562 INFO:teuthology.orchestra.run.smithi156.stdout:506 Cannot talk to daemon 2024-01-16T04:46:17.575 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-16T04:46:17.588 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-16T04:46:17.653 INFO:teuthology.orchestra.run.smithi156.stderr:bash: ntpq: command not found 2024-01-16T04:46:17.656 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T04:46:17.656 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-01-16T04:46:17.656 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T04:46:17.656 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T04:46:17.656 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T04:46:17.656 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T04:46:17.657 INFO:teuthology.run_tasks:Running task install... 2024-01-16T04:46:17.666 DEBUG:teuthology.task.install:project ceph 2024-01-16T04:46:17.666 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f'}} 2024-01-16T04:46:17.666 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f'} 2024-01-16T04:46:17.666 INFO:teuthology.task.install:Using flavor: default 2024-01-16T04:46:17.671 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-16T04:46:17.672 INFO:teuthology.task.install:extra packages: [] 2024-01-16T04:46:17.672 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f', 'tag': None, 'wait_for_package': False} 2024-01-16T04:46:17.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a70aebef70df6ec2d957347ccbfc484e0573dd5f 2024-01-16T04:46:17.817 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-01-11-1300-pacific/a70aebef70df6ec2d957347ccbfc484e0573dd5f/centos/8/flavors/default/ 2024-01-16T04:46:17.817 INFO:teuthology.task.install.rpm:Package version is 16.2.14-449.ga70aebef 2024-01-16T04:46:17.991 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-01-11-1300-pacific/a70aebef70df6ec2d957347ccbfc484e0573dd5f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-01-11-1300-pacific/a70aebef70df6ec2d957347ccbfc484e0573dd5f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-01-11-1300-pacific/a70aebef70df6ec2d957347ccbfc484e0573dd5f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-16T04:46:17.991 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:46:17.991 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-16T04:46:18.019 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-16T04:46:18.019 DEBUG:teuthology.orchestra.run.smithi156:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a70aebef70df6ec2d957347ccbfc484e0573dd5f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-16T04:46:18.087 DEBUG:teuthology.orchestra.run.smithi156:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-16T04:46:18.164 DEBUG:teuthology.orchestra.run.smithi156:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-16T04:46:18.234 INFO:teuthology.orchestra.run.smithi156.stdout:check_obsoletes = 1 2024-01-16T04:46:18.235 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum clean all 2024-01-16T04:46:18.504 INFO:teuthology.orchestra.run.smithi156.stdout:76 files removed 2024-01-16T04:46:18.532 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw 2024-01-16T04:46:19.203 INFO:teuthology.orchestra.run.smithi156.stdout:ceph packages for x86_64 289 kB/s | 76 kB 00:00 2024-01-16T04:46:19.571 INFO:teuthology.orchestra.run.smithi156.stdout:ceph noarch packages 47 kB/s | 15 kB 00:00 2024-01-16T04:46:20.021 INFO:teuthology.orchestra.run.smithi156.stdout:ceph source packages 3.4 kB/s | 1.4 kB 00:00 2024-01-16T04:46:24.479 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - AppStream 7.9 MB/s | 35 MB 00:04 2024-01-16T04:46:33.221 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - BaseOS 25 MB/s | 58 MB 00:02 2024-01-16T04:46:41.896 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - Extras 42 kB/s | 18 kB 00:00 2024-01-16T04:46:42.613 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2024-01-16T04:46:44.839 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 8 - PowerTools 3.2 MB/s | 6.7 MB 00:02 2024-01-16T04:46:46.391 INFO:teuthology.orchestra.run.smithi156.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2024-01-16T04:46:47.477 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2024-01-16T04:46:51.425 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 275 kB/s | 24 kB 00:00 2024-01-16T04:46:56.176 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:46:56.179 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:46:56.179 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-01-16T04:46:56.179 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:46:56.179 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:46:56.179 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw x86_64 2:16.2.14-449.ga70aebef.el8 ceph 12 M 2024-01-16T04:46:56.179 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2024-01-16T04:46:56.180 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 x86_64 2:16.2.14-449.ga70aebef.el8 ceph 3.7 M 2024-01-16T04:46:56.180 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1 x86_64 2:16.2.14-449.ga70aebef.el8 ceph 4.2 M 2024-01-16T04:46:56.180 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-01-16T04:46:56.180 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base x86_64 2:16.2.14-449.ga70aebef.el8 ceph 6.1 M 2024-01-16T04:46:56.180 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common x86_64 2:16.2.14-449.ga70aebef.el8 ceph 24 M 2024-01-16T04:46:56.180 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux x86_64 2:16.2.14-449.ga70aebef.el8 ceph 26 k 2024-01-16T04:46:56.180 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-16T04:46:56.181 INFO:teuthology.orchestra.run.smithi156.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-16T04:46:56.181 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2 x86_64 2:16.2.14-449.ga70aebef.el8 ceph 825 k 2024-01-16T04:46:56.181 INFO:teuthology.orchestra.run.smithi156.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-16T04:46:56.181 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2024-01-16T04:46:56.181 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1 x86_64 2:16.2.14-449.ga70aebef.el8 ceph 494 k 2024-01-16T04:46:56.181 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-16T04:46:56.181 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 x86_64 2:16.2.14-449.ga70aebef.el8 ceph 3.8 M 2024-01-16T04:46:56.181 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-16T04:46:56.182 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-16T04:46:56.182 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse x86_64 2:16.2.14-449.ga70aebef.el8 ceph 45 k 2024-01-16T04:46:56.182 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common x86_64 2:16.2.14-449.ga70aebef.el8 ceph 99 k 2024-01-16T04:46:56.182 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs x86_64 2:16.2.14-449.ga70aebef.el8 ceph 215 k 2024-01-16T04:46:56.182 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-16T04:46:56.182 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados x86_64 2:16.2.14-449.ga70aebef.el8 ceph 389 k 2024-01-16T04:46:56.182 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd x86_64 2:16.2.14-449.ga70aebef.el8 ceph 369 k 2024-01-16T04:46:56.182 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw x86_64 2:16.2.14-449.ga70aebef.el8 ceph 114 k 2024-01-16T04:46:56.182 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:46:56.182 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:46:56.183 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:46:56.183 INFO:teuthology.orchestra.run.smithi156.stdout:Install 21 Packages 2024-01-16T04:46:56.183 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 2 Packages 2024-01-16T04:46:56.183 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:46:56.183 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 58 M 2024-01-16T04:46:56.183 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:46:57.449 INFO:teuthology.orchestra.run.smithi156.stdout:(1/23): ceph-base-16.2.14-449.ga70aebef.el8.x86 7.0 MB/s | 6.1 MB 00:00 2024-01-16T04:46:57.577 INFO:teuthology.orchestra.run.smithi156.stdout:(2/23): ceph-radosgw-16.2.14-449.ga70aebef.el8. 12 MB/s | 12 MB 00:00 2024-01-16T04:46:57.602 INFO:teuthology.orchestra.run.smithi156.stdout:(3/23): ceph-selinux-16.2.14-449.ga70aebef.el8. 167 kB/s | 26 kB 00:00 2024-01-16T04:46:57.652 INFO:teuthology.orchestra.run.smithi156.stdout:(4/23): libcephfs2-16.2.14-449.ga70aebef.el8.x8 11 MB/s | 825 kB 00:00 2024-01-16T04:46:57.677 INFO:teuthology.orchestra.run.smithi156.stdout:(5/23): libradosstriper1-16.2.14-449.ga70aebef. 6.4 MB/s | 494 kB 00:00 2024-01-16T04:46:57.744 INFO:teuthology.orchestra.run.smithi156.stdout:(6/23): python3-ceph-argparse-16.2.14-449.ga70a 676 kB/s | 45 kB 00:00 2024-01-16T04:46:57.811 INFO:teuthology.orchestra.run.smithi156.stdout:(7/23): python3-ceph-common-16.2.14-449.ga70aeb 1.4 MB/s | 99 kB 00:00 2024-01-16T04:46:57.887 INFO:teuthology.orchestra.run.smithi156.stdout:(8/23): librgw2-16.2.14-449.ga70aebef.el8.x86_6 16 MB/s | 3.8 MB 00:00 2024-01-16T04:46:57.912 INFO:teuthology.orchestra.run.smithi156.stdout:(9/23): python3-cephfs-16.2.14-449.ga70aebef.el 2.1 MB/s | 215 kB 00:00 2024-01-16T04:46:57.937 INFO:teuthology.orchestra.run.smithi156.stdout:(10/23): python3-rados-16.2.14-449.ga70aebef.el 7.6 MB/s | 389 kB 00:00 2024-01-16T04:46:57.971 INFO:teuthology.orchestra.run.smithi156.stdout:(11/23): python3-rbd-16.2.14-449.ga70aebef.el8. 6.2 MB/s | 369 kB 00:00 2024-01-16T04:46:57.996 INFO:teuthology.orchestra.run.smithi156.stdout:(12/23): python3-rgw-16.2.14-449.ga70aebef.el8. 1.9 MB/s | 114 kB 00:00 2024-01-16T04:46:58.222 INFO:teuthology.orchestra.run.smithi156.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.1 MB/s | 259 kB 00:00 2024-01-16T04:46:58.330 INFO:teuthology.orchestra.run.smithi156.stdout:(14/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.8 MB/s | 677 kB 00:00 2024-01-16T04:46:58.381 INFO:teuthology.orchestra.run.smithi156.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 276 kB/s | 44 kB 00:00 2024-01-16T04:46:58.406 INFO:teuthology.orchestra.run.smithi156.stdout:(16/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 622 kB/s | 47 kB 00:00 2024-01-16T04:46:59.098 INFO:teuthology.orchestra.run.smithi156.stdout:(17/23): leveldb-1.22-1.el8.x86_64.rpm 261 kB/s | 181 kB 00:00 2024-01-16T04:46:59.307 INFO:teuthology.orchestra.run.smithi156.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 282 kB/s | 59 kB 00:00 2024-01-16T04:46:59.340 INFO:teuthology.orchestra.run.smithi156.stdout:(19/23): gperftools-libs-2.7-9.el8.x86_64.rpm 319 kB/s | 306 kB 00:00 2024-01-16T04:46:59.516 INFO:teuthology.orchestra.run.smithi156.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 359 kB/s | 75 kB 00:00 2024-01-16T04:46:59.682 INFO:teuthology.orchestra.run.smithi156.stdout:(21/23): librados2-16.2.14-449.ga70aebef.el8.x8 11 MB/s | 3.7 MB 00:00 2024-01-16T04:46:59.924 INFO:teuthology.orchestra.run.smithi156.stdout:(22/23): librbd1-16.2.14-449.ga70aebef.el8.x86_ 10 MB/s | 4.2 MB 00:00 2024-01-16T04:47:02.124 INFO:teuthology.orchestra.run.smithi156.stdout:(23/23): ceph-common-16.2.14-449.ga70aebef.el8. 4.4 MB/s | 24 MB 00:05 2024-01-16T04:47:02.127 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:47:02.127 INFO:teuthology.orchestra.run.smithi156.stdout:Total 9.7 MB/s | 58 MB 00:05 2024-01-16T04:47:02.172 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:47:02.220 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:47:02.220 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:47:02.572 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:47:02.575 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:47:03.687 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:47:03.904 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-01-16T04:47:04.334 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-01-16T04:47:04.732 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-01-16T04:47:04.926 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librados2-2:16.2.14-449.ga70aebef.el8.x86_64 3/25 2024-01-16T04:47:05.024 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:16.2.14-449.ga70aebef.el8.x86_64 3/25 2024-01-16T04:47:05.308 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rados-2:16.2.14-449.ga70aebef.el8.x86_64 4/25 2024-01-16T04:47:05.476 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs2-2:16.2.14-449.ga70aebef.el8.x86_64 5/25 2024-01-16T04:47:05.897 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs2-2:16.2.14-449.ga70aebef.el8.x86_64 5/25 2024-01-16T04:47:06.092 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librbd1-2:16.2.14-449.ga70aebef.el8.x86_64 6/25 2024-01-16T04:47:06.138 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:16.2.14-449.ga70aebef.el8.x86_64 6/25 2024-01-16T04:47:06.344 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2024-01-16T04:47:06.857 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-01-16T04:47:07.034 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librgw2-2:16.2.14-449.ga70aebef.el8.x86_64 9/25 2024-01-16T04:47:07.103 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librgw2-2:16.2.14-449.ga70aebef.el8.x86_64 9/25 2024-01-16T04:47:07.348 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-01-16T04:47:07.523 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-argparse-2:16.2.14-449.ga70aebef.el8. 11/25 2024-01-16T04:47:07.721 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cephfs-2:16.2.14-449.ga70aebef.el8.x86_64 12/25 2024-01-16T04:47:07.948 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rgw-2:16.2.14-449.ga70aebef.el8.x86_64 13/25 2024-01-16T04:47:08.181 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rbd-2:16.2.14-449.ga70aebef.el8.x86_64 14/25 2024-01-16T04:47:08.350 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libradosstriper1-2:16.2.14-449.ga70aebef.el8.x86_6 15/25 2024-01-16T04:47:08.424 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libradosstriper1-2:16.2.14-449.ga70aebef.el8.x86_6 15/25 2024-01-16T04:47:08.610 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-01-16T04:47:08.825 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-01-16T04:47:09.047 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-01-16T04:47:09.259 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-common-2:16.2.14-449.ga70aebef.el8.x8 19/25 2024-01-16T04:47:11.853 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 20/25 2024-01-16T04:47:12.084 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 20/25 2024-01-16T04:47:12.774 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 20/25 2024-01-16T04:47:13.134 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-base-2:16.2.14-449.ga70aebef.el8.x86_64 21/25 2024-01-16T04:47:13.244 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-base-2:16.2.14-449.ga70aebef.el8.x86_64 21/25 2024-01-16T04:47:13.383 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-selinux-2:16.2.14-449.ga70aebef.el8.x86_64 22/25 2024-01-16T04:47:32.758 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-selinux-2:16.2.14-449.ga70aebef.el8.x86_64 22/25 2024-01-16T04:47:32.758 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /sys 2024-01-16T04:47:32.758 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /proc 2024-01-16T04:47:32.758 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /dev 2024-01-16T04:47:32.758 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /run 2024-01-16T04:47:32.758 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /mnt 2024-01-16T04:47:32.758 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /var/tmp 2024-01-16T04:47:32.758 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /home 2024-01-16T04:47:32.758 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /tmp 2024-01-16T04:47:32.758 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /dev 2024-01-16T04:47:32.758 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:47:33.922 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-radosgw-2:16.2.14-449.ga70aebef.el8.x86_64 23/25 2024-01-16T04:47:33.998 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-449.ga70aebef.el8.x86_64 23/25 2024-01-16T04:47:33.998 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-01-16T04:47:34.215 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-01-16T04:47:34.215 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-01-16T04:47:34.917 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-01-16T04:47:34.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-base-2:16.2.14-449.ga70aebef.el8.x86_64 1/25 2024-01-16T04:47:34.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 2/25 2024-01-16T04:47:34.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-radosgw-2:16.2.14-449.ga70aebef.el8.x86_64 3/25 2024-01-16T04:47:34.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-selinux-2:16.2.14-449.ga70aebef.el8.x86_64 4/25 2024-01-16T04:47:34.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs2-2:16.2.14-449.ga70aebef.el8.x86_64 5/25 2024-01-16T04:47:34.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libradosstriper1-2:16.2.14-449.ga70aebef.el8.x86_6 6/25 2024-01-16T04:47:34.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librgw2-2:16.2.14-449.ga70aebef.el8.x86_64 7/25 2024-01-16T04:47:34.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-argparse-2:16.2.14-449.ga70aebef.el8. 8/25 2024-01-16T04:47:34.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-common-2:16.2.14-449.ga70aebef.el8.x8 9/25 2024-01-16T04:47:34.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cephfs-2:16.2.14-449.ga70aebef.el8.x86_64 10/25 2024-01-16T04:47:34.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rados-2:16.2.14-449.ga70aebef.el8.x86_64 11/25 2024-01-16T04:47:34.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rbd-2:16.2.14-449.ga70aebef.el8.x86_64 12/25 2024-01-16T04:47:34.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rgw-2:16.2.14-449.ga70aebef.el8.x86_64 13/25 2024-01-16T04:47:34.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-01-16T04:47:34.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-01-16T04:47:34.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-01-16T04:47:34.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2024-01-16T04:47:34.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-01-16T04:47:34.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-01-16T04:47:34.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-01-16T04:47:34.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-01-16T04:47:34.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:16.2.14-449.ga70aebef.el8.x86_64 22/25 2024-01-16T04:47:34.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-01-16T04:47:34.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:16.2.14-449.ga70aebef.el8.x86_64 24/25 2024-01-16T04:47:36.119 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-01-16T04:47:36.119 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:47:36.119 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2024-01-16T04:47:36.119 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:36.119 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:36.119 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:47:36.119 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:36.119 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:36.119 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:36.120 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:36.120 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-16T04:47:36.120 INFO:teuthology.orchestra.run.smithi156.stdout: leveldb-1.22-1.el8.x86_64 2024-01-16T04:47:36.120 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:36.120 INFO:teuthology.orchestra.run.smithi156.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-16T04:47:36.120 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq-0.9.0-4.el8.x86_64 2024-01-16T04:47:36.120 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:36.120 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-16T04:47:36.120 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:36.120 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-16T04:47:36.120 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-16T04:47:36.120 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:36.120 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:36.120 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:36.121 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-16T04:47:36.121 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:36.121 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:36.121 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:36.121 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:47:36.121 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:47:36.351 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-test 2024-01-16T04:47:37.213 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:00:46 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:47:38.935 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:47:38.936 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:47:38.936 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-01-16T04:47:38.936 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:47:38.936 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:47:38.936 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test x86_64 2:16.2.14-449.ga70aebef.el8 ceph 52 M 2024-01-16T04:47:38.936 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-01-16T04:47:38.936 INFO:teuthology.orchestra.run.smithi156.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-16T04:47:38.936 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite x86_64 2:16.2.14-449.ga70aebef.el8 ceph 174 k 2024-01-16T04:47:38.936 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2024-01-16T04:47:38.936 INFO:teuthology.orchestra.run.smithi156.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-16T04:47:38.936 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-16T04:47:38.936 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:47:38.937 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:47:38.937 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:47:38.937 INFO:teuthology.orchestra.run.smithi156.stdout:Install 6 Packages 2024-01-16T04:47:38.937 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:47:38.937 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 53 M 2024-01-16T04:47:38.937 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 196 M 2024-01-16T04:47:38.937 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:47:39.207 INFO:teuthology.orchestra.run.smithi156.stdout:(1/6): libcephsqlite-16.2.14-449.ga70aebef.el8. 894 kB/s | 174 kB 00:00 2024-01-16T04:47:39.515 INFO:teuthology.orchestra.run.smithi156.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 404 kB/s | 203 kB 00:00 2024-01-16T04:47:39.540 INFO:teuthology.orchestra.run.smithi156.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 562 kB/s | 187 kB 00:00 2024-01-16T04:47:39.716 INFO:teuthology.orchestra.run.smithi156.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 403 kB/s | 71 kB 00:00 2024-01-16T04:47:39.999 INFO:teuthology.orchestra.run.smithi156.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 668 kB/s | 323 kB 00:00 2024-01-16T04:47:41.549 INFO:teuthology.orchestra.run.smithi156.stdout:(6/6): ceph-test-16.2.14-449.ga70aebef.el8.x86_ 20 MB/s | 52 MB 00:02 2024-01-16T04:47:41.551 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:47:41.552 INFO:teuthology.orchestra.run.smithi156.stdout:Total 20 MB/s | 53 MB 00:02 2024-01-16T04:47:41.596 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:47:41.616 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:47:41.616 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:47:41.936 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:47:41.936 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:47:42.429 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:47:42.628 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-16T04:47:42.839 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-16T04:47:42.978 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2024-01-16T04:47:43.055 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2024-01-16T04:47:43.268 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-16T04:47:43.420 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephsqlite-2:16.2.14-449.ga70aebef.el8.x86_64 5/6 2024-01-16T04:47:48.022 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephsqlite-2:16.2.14-449.ga70aebef.el8.x86_64 5/6 2024-01-16T04:47:48.276 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-test-2:16.2.14-449.ga70aebef.el8.x86_64 6/6 2024-01-16T04:47:48.570 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-test-2:16.2.14-449.ga70aebef.el8.x86_64 6/6 2024-01-16T04:47:48.570 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-test-2:16.2.14-449.ga70aebef.el8.x86_64 1/6 2024-01-16T04:47:48.570 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephsqlite-2:16.2.14-449.ga70aebef.el8.x86_64 2/6 2024-01-16T04:47:48.570 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-16T04:47:48.570 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2024-01-16T04:47:48.571 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-16T04:47:49.304 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-16T04:47:49.305 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:47:49.305 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:47:49.305 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:49.305 INFO:teuthology.orchestra.run.smithi156.stdout: jq-1.6-8.el8.x86_64 2024-01-16T04:47:49.305 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:47:49.305 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma-6.8.2-2.el8.x86_64 2024-01-16T04:47:49.305 INFO:teuthology.orchestra.run.smithi156.stdout: socat-1.7.4.1-1.el8.x86_64 2024-01-16T04:47:49.305 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-01-16T04:47:49.305 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:47:49.305 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:47:49.527 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph 2024-01-16T04:47:50.387 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:00:59 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:47:52.101 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:47:52.105 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:47:52.105 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-01-16T04:47:52.105 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:47:52.105 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:47:52.105 INFO:teuthology.orchestra.run.smithi156.stdout: ceph x86_64 2:16.2.14-449.ga70aebef.el8 ceph 6.4 k 2024-01-16T04:47:52.105 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-01-16T04:47:52.105 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds x86_64 2:16.2.14-449.ga70aebef.el8 ceph 2.4 M 2024-01-16T04:47:52.105 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr x86_64 2:16.2.14-449.ga70aebef.el8 ceph 1.6 M 2024-01-16T04:47:52.105 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 245 k 2024-01-16T04:47:52.105 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon x86_64 2:16.2.14-449.ga70aebef.el8 ceph 5.0 M 2024-01-16T04:47:52.106 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd x86_64 2:16.2.14-449.ga70aebef.el8 ceph 18 M 2024-01-16T04:47:52.106 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-16T04:47:52.106 INFO:teuthology.orchestra.run.smithi156.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-16T04:47:52.106 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-16T04:47:52.106 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-16T04:47:52.106 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-16T04:47:52.106 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-16T04:47:52.106 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-16T04:47:52.106 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-16T04:47:52.106 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-16T04:47:52.106 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-16T04:47:52.106 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-16T04:47:52.107 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-16T04:47:52.107 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-16T04:47:52.107 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-16T04:47:52.107 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-16T04:47:52.107 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-16T04:47:52.107 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-16T04:47:52.107 INFO:teuthology.orchestra.run.smithi156.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-16T04:47:52.107 INFO:teuthology.orchestra.run.smithi156.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-16T04:47:52.107 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-16T04:47:52.107 INFO:teuthology.orchestra.run.smithi156.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-16T04:47:52.107 INFO:teuthology.orchestra.run.smithi156.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-16T04:47:52.107 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-16T04:47:52.107 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-16T04:47:52.107 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-16T04:47:52.108 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-16T04:47:52.108 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:47:52.108 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:47:52.108 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:47:52.108 INFO:teuthology.orchestra.run.smithi156.stdout:Install 32 Packages 2024-01-16T04:47:52.108 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:47:52.108 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 31 M 2024-01-16T04:47:52.109 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 101 M 2024-01-16T04:47:52.109 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:47:52.685 INFO:teuthology.orchestra.run.smithi156.stdout:(1/32): ceph-16.2.14-449.ga70aebef.el8.x86_64.r 50 kB/s | 6.4 kB 00:00 2024-01-16T04:47:52.919 INFO:teuthology.orchestra.run.smithi156.stdout:(2/32): ceph-mds-16.2.14-449.ga70aebef.el8.x86_ 6.7 MB/s | 2.4 MB 00:00 2024-01-16T04:47:52.952 INFO:teuthology.orchestra.run.smithi156.stdout:(3/32): ceph-mgr-16.2.14-449.ga70aebef.el8.x86_ 4.1 MB/s | 1.6 MB 00:00 2024-01-16T04:47:53.044 INFO:teuthology.orchestra.run.smithi156.stdout:(4/32): ceph-mgr-modules-core-16.2.14-449.ga70a 2.6 MB/s | 245 kB 00:00 2024-01-16T04:47:53.478 INFO:teuthology.orchestra.run.smithi156.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 362 kB/s | 157 kB 00:00 2024-01-16T04:47:53.603 INFO:teuthology.orchestra.run.smithi156.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 311 kB/s | 39 kB 00:00 2024-01-16T04:47:53.846 INFO:teuthology.orchestra.run.smithi156.stdout:(7/32): ceph-osd-16.2.14-449.ga70aebef.el8.x86_ 20 MB/s | 18 MB 00:00 2024-01-16T04:47:53.871 INFO:teuthology.orchestra.run.smithi156.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 388 kB/s | 103 kB 00:00 2024-01-16T04:47:53.946 INFO:teuthology.orchestra.run.smithi156.stdout:(9/32): ceph-mon-16.2.14-449.ga70aebef.el8.x86_ 4.0 MB/s | 5.0 MB 00:01 2024-01-16T04:47:54.163 INFO:teuthology.orchestra.run.smithi156.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 180 kB/s | 57 kB 00:00 2024-01-16T04:47:54.263 INFO:teuthology.orchestra.run.smithi156.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 785 kB/s | 249 kB 00:00 2024-01-16T04:47:54.347 INFO:teuthology.orchestra.run.smithi156.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.3 MB/s | 109 kB 00:00 2024-01-16T04:47:54.373 INFO:teuthology.orchestra.run.smithi156.stdout:(13/32): python3-werkzeug-0.12.2-4.el8.noarch.r 912 kB/s | 457 kB 00:00 2024-01-16T04:47:54.539 INFO:teuthology.orchestra.run.smithi156.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.7 MB/s | 641 kB 00:00 2024-01-16T04:47:54.540 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm: Status code: 403 for http://mirror.usi.edu/pub/epel/8/Everything/x86_64/Packages/p/python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm (IP: 192.206.9.160) 2024-01-16T04:47:54.540 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] python3-beautifulsoup4-4.6.3-2.el8.1.noarch.rpm: Status code: 403 for http://mirror.usi.edu/pub/epel/8/Everything/x86_64/Packages/p/python3-beautifulsoup4-4.6.3-2.el8.1.noarch.rpm (IP: 192.206.9.160) 2024-01-16T04:47:54.612 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] python3-cheroot-8.5.2-1.el8.noarch.rpm: Status code: 403 for http://mirror.usi.edu/pub/epel/8/Everything/x86_64/Packages/p/python3-cheroot-8.5.2-1.el8.noarch.rpm (IP: 192.206.9.160) 2024-01-16T04:47:54.757 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm: Status code: 403 for https://mirror.usi.edu/pub/epel/8/Everything/x86_64/Packages/p/python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm (IP: 192.206.9.160) 2024-01-16T04:47:54.763 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] python3-beautifulsoup4-4.6.3-2.el8.1.noarch.rpm: Status code: 403 for https://mirror.usi.edu/pub/epel/8/Everything/x86_64/Packages/p/python3-beautifulsoup4-4.6.3-2.el8.1.noarch.rpm (IP: 192.206.9.160) 2024-01-16T04:47:54.820 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] python3-cheroot-8.5.2-1.el8.noarch.rpm: Status code: 403 for https://mirror.usi.edu/pub/epel/8/Everything/x86_64/Packages/p/python3-cheroot-8.5.2-1.el8.noarch.rpm (IP: 192.206.9.160) 2024-01-16T04:47:54.881 INFO:teuthology.orchestra.run.smithi156.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 82 kB/s | 44 kB 00:00 2024-01-16T04:47:54.931 INFO:teuthology.orchestra.run.smithi156.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 331 kB/s | 185 kB 00:00 2024-01-16T04:47:54.941 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] python3-cherrypy-18.4.0-1.el8.noarch.rpm: Status code: 403 for https://mirror.usi.edu/pub/epel/8/Everything/x86_64/Packages/p/python3-cherrypy-18.4.0-1.el8.noarch.rpm (IP: 192.206.9.160) 2024-01-16T04:47:54.990 INFO:teuthology.orchestra.run.smithi156.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 383 kB/s | 173 kB 00:00 2024-01-16T04:47:54.990 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] python3-jaraco-6.2-6.el8.noarch.rpm: Status code: 403 for https://mirror.usi.edu/pub/epel/8/Everything/x86_64/Packages/p/python3-jaraco-6.2-6.el8.noarch.rpm (IP: 192.206.9.160) 2024-01-16T04:47:54.999 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] python3-cherrypy-18.4.0-1.el8.noarch.rpm: Status code: 403 for http://mirror.usi.edu/pub/epel/8/Everything/x86_64/Packages/p/python3-cherrypy-18.4.0-1.el8.noarch.rpm (IP: 192.206.9.160) 2024-01-16T04:47:55.044 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] python3-jaraco-functools-2.0-4.el8.noarch.rpm: Status code: 403 for http://mirror.usi.edu/pub/epel/8/Everything/x86_64/Packages/p/python3-jaraco-functools-2.0-4.el8.noarch.rpm (IP: 192.206.9.160) 2024-01-16T04:47:55.091 INFO:teuthology.orchestra.run.smithi156.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2024-01-16T04:47:55.091 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] python3-jaraco-6.2-6.el8.noarch.rpm: Status code: 403 for http://mirror.usi.edu/pub/epel/8/Everything/x86_64/Packages/p/python3-jaraco-6.2-6.el8.noarch.rpm (IP: 192.206.9.160) 2024-01-16T04:47:55.133 INFO:teuthology.orchestra.run.smithi156.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.5 MB/s | 384 kB 00:00 2024-01-16T04:47:55.158 INFO:teuthology.orchestra.run.smithi156.stdout:(20/32): python3-jaraco-6.2-6.el8.noarch.rpm 49 kB/s | 11 kB 00:00 2024-01-16T04:47:55.183 INFO:teuthology.orchestra.run.smithi156.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 528 kB/s | 49 kB 00:00 2024-01-16T04:47:55.208 INFO:teuthology.orchestra.run.smithi156.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 778 kB/s | 59 kB 00:00 2024-01-16T04:47:55.256 INFO:teuthology.orchestra.run.smithi156.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 216 kB/s | 16 kB 00:00 2024-01-16T04:47:55.284 INFO:teuthology.orchestra.run.smithi156.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 248 kB/s | 19 kB 00:00 2024-01-16T04:47:55.309 INFO:teuthology.orchestra.run.smithi156.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 450 kB/s | 24 kB 00:00 2024-01-16T04:47:55.335 INFO:teuthology.orchestra.run.smithi156.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 578 kB/s | 29 kB 00:00 2024-01-16T04:47:55.360 INFO:teuthology.orchestra.run.smithi156.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 544 kB/s | 27 kB 00:00 2024-01-16T04:47:55.385 INFO:teuthology.orchestra.run.smithi156.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2024-01-16T04:47:55.411 INFO:teuthology.orchestra.run.smithi156.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2024-01-16T04:47:55.436 INFO:teuthology.orchestra.run.smithi156.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.3 MB/s | 251 kB 00:00 2024-01-16T04:47:55.453 INFO:teuthology.orchestra.run.smithi156.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.2 MB/s | 85 kB 00:00 2024-01-16T04:47:55.470 INFO:teuthology.orchestra.run.smithi156.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 392 kB/s | 23 kB 00:00 2024-01-16T04:47:55.472 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:47:55.473 INFO:teuthology.orchestra.run.smithi156.stdout:Total 9.3 MB/s | 31 MB 00:03 2024-01-16T04:47:55.558 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:47:55.597 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:47:55.597 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:47:55.921 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:47:55.924 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:47:57.050 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:47:57.281 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-16T04:47:57.484 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-16T04:47:57.700 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-16T04:47:57.940 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-16T04:47:58.147 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-16T04:47:58.311 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-16T04:47:58.512 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-16T04:47:58.699 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-16T04:47:58.882 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-16T04:47:59.103 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-16T04:47:59.358 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-16T04:47:59.573 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-16T04:47:59.965 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-16T04:48:00.246 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-16T04:48:00.450 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-16T04:48:00.716 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-16T04:48:00.909 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-16T04:48:01.243 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-16T04:48:01.484 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-16T04:48:01.703 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-16T04:48:01.897 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-16T04:48:02.122 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-16T04:48:02.385 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-16T04:48:02.609 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-16T04:48:02.915 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-16T04:48:03.827 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-16T04:48:04.204 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-449.ga70aebef.el8. 27/32 2024-01-16T04:48:04.384 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-2:16.2.14-449.ga70aebef.el8.x86_64 28/32 2024-01-16T04:48:05.874 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-2:16.2.14-449.ga70aebef.el8.x86_64 28/32 2024-01-16T04:48:06.083 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-osd-2:16.2.14-449.ga70aebef.el8.x86_64 29/32 2024-01-16T04:48:06.509 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-osd-2:16.2.14-449.ga70aebef.el8.x86_64 29/32 2024-01-16T04:48:06.683 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mon-2:16.2.14-449.ga70aebef.el8.x86_64 30/32 2024-01-16T04:48:06.900 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mon-2:16.2.14-449.ga70aebef.el8.x86_64 30/32 2024-01-16T04:48:07.091 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mds-2:16.2.14-449.ga70aebef.el8.x86_64 31/32 2024-01-16T04:48:07.109 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mds-2:16.2.14-449.ga70aebef.el8.x86_64 31/32 2024-01-16T04:48:07.257 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-2:16.2.14-449.ga70aebef.el8.x86_64 32/32 2024-01-16T04:48:07.645 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-2:16.2.14-449.ga70aebef.el8.x86_64 32/32 2024-01-16T04:48:07.645 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-2:16.2.14-449.ga70aebef.el8.x86_64 1/32 2024-01-16T04:48:07.645 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mds-2:16.2.14-449.ga70aebef.el8.x86_64 2/32 2024-01-16T04:48:07.645 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-2:16.2.14-449.ga70aebef.el8.x86_64 3/32 2024-01-16T04:48:07.645 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mon-2:16.2.14-449.ga70aebef.el8.x86_64 4/32 2024-01-16T04:48:07.645 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-osd-2:16.2.14-449.ga70aebef.el8.x86_64 5/32 2024-01-16T04:48:07.645 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-449.ga70aebef.el8. 6/32 2024-01-16T04:48:07.645 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-16T04:48:07.645 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-16T04:48:07.645 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-16T04:48:07.645 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-16T04:48:07.645 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-16T04:48:07.646 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-16T04:48:07.646 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-16T04:48:07.646 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-16T04:48:07.646 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-16T04:48:07.646 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-16T04:48:07.646 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-16T04:48:07.646 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-16T04:48:07.646 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-16T04:48:07.646 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-16T04:48:07.646 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-16T04:48:07.646 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-16T04:48:07.646 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-16T04:48:07.646 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-16T04:48:07.646 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-16T04:48:07.647 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-16T04:48:07.647 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-16T04:48:07.647 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-16T04:48:07.647 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-16T04:48:07.647 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-16T04:48:07.647 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-16T04:48:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-16T04:48:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:08.778 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:48:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:48:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:48:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:48:08.778 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:48:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:48:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:48:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-16T04:48:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-16T04:48:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-16T04:48:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-16T04:48:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-16T04:48:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-16T04:48:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-16T04:48:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-16T04:48:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-16T04:48:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-16T04:48:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-16T04:48:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-16T04:48:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-16T04:48:08.779 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend-2.6-1.el8.noarch 2024-01-16T04:48:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-16T04:48:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-16T04:48:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-16T04:48:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-16T04:48:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-16T04:48:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-16T04:48:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-16T04:48:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-16T04:48:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-16T04:48:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-16T04:48:08.780 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-16T04:48:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-16T04:48:08.781 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:08.781 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:48:09.037 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-base 2024-01-16T04:48:09.894 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:18 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:48:11.370 INFO:teuthology.orchestra.run.smithi156.stdout:Package ceph-base-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:48:11.604 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:48:11.605 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-01-16T04:48:11.606 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:48:11.709 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install cephadm 2024-01-16T04:48:12.563 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:21 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:48:14.278 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:48:14.278 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:48:14.278 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-01-16T04:48:14.278 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:48:14.278 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:48:14.278 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 85 k 2024-01-16T04:48:14.278 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:14.279 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:48:14.279 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:48:14.279 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-01-16T04:48:14.279 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:14.279 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 85 k 2024-01-16T04:48:14.279 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 352 k 2024-01-16T04:48:14.279 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:48:14.621 INFO:teuthology.orchestra.run.smithi156.stdout:cephadm-16.2.14-449.ga70aebef.el8.noarch.rpm 250 kB/s | 85 kB 00:00 2024-01-16T04:48:14.621 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:48:14.622 INFO:teuthology.orchestra.run.smithi156.stdout:Total 249 kB/s | 85 kB 00:00 2024-01-16T04:48:14.622 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:48:14.630 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:48:14.630 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:48:14.632 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:48:14.635 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:48:14.682 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:48:15.188 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:16.2.14-449.ga70aebef.el8.noarch 1/1 2024-01-16T04:48:15.370 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephadm-2:16.2.14-449.ga70aebef.el8.noarch 1/1 2024-01-16T04:48:15.602 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:16.2.14-449.ga70aebef.el8.noarch 1/1 2024-01-16T04:48:16.312 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephadm-2:16.2.14-449.ga70aebef.el8.noarch 1/1 2024-01-16T04:48:16.313 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:16.313 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:48:16.313 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:48:16.313 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:16.313 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:48:16.512 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-immutable-object-cache 2024-01-16T04:48:17.378 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:26 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:48:19.105 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:48:19.105 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:48:19.105 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-01-16T04:48:19.105 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:48:19.106 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:48:19.106 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-449.ga70aebef.el8 ceph 161 k 2024-01-16T04:48:19.106 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:19.106 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:48:19.106 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:48:19.106 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-01-16T04:48:19.106 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:19.106 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 161 k 2024-01-16T04:48:19.107 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 470 k 2024-01-16T04:48:19.107 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:48:19.426 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-immutable-object-cache-16.2.14-449.ga70aeb 505 kB/s | 161 kB 00:00 2024-01-16T04:48:19.427 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:48:19.427 INFO:teuthology.orchestra.run.smithi156.stdout:Total 503 kB/s | 161 kB 00:00 2024-01-16T04:48:19.427 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:48:19.437 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:48:19.437 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:48:19.485 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:48:19.488 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:48:19.594 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:48:19.764 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-449.ga70aebef. 1/1 2024-01-16T04:48:20.796 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-449.ga70aebef. 1/1 2024-01-16T04:48:21.799 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-449.ga70aebef. 1/1 2024-01-16T04:48:21.800 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:21.800 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:48:21.800 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:48:21.800 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:21.800 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:48:22.035 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr 2024-01-16T04:48:22.894 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:31 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:48:24.355 INFO:teuthology.orchestra.run.smithi156.stdout:Package ceph-mgr-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:48:24.591 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:48:24.591 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-01-16T04:48:24.592 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:48:24.692 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-dashboard 2024-01-16T04:48:25.541 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:34 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:48:27.246 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:48:27.247 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:48:27.248 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-01-16T04:48:27.248 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:48:27.248 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:48:27.248 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 3.1 M 2024-01-16T04:48:27.248 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-01-16T04:48:27.248 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 24 k 2024-01-16T04:48:27.248 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 15 k 2024-01-16T04:48:27.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-16T04:48:27.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-16T04:48:27.248 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-16T04:48:27.248 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:27.248 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:48:27.248 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:48:27.249 INFO:teuthology.orchestra.run.smithi156.stdout:Install 6 Packages 2024-01-16T04:48:27.249 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:27.249 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 3.4 M 2024-01-16T04:48:27.250 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 22 M 2024-01-16T04:48:27.250 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:48:27.855 INFO:teuthology.orchestra.run.smithi156.stdout:(1/6): ceph-grafana-dashboards-16.2.14-449.ga70 168 kB/s | 24 kB 00:00 2024-01-16T04:48:27.989 INFO:teuthology.orchestra.run.smithi156.stdout:(2/6): python3-jwt-1.6.1-2.el8.noarch.rpm 321 kB/s | 43 kB 00:00 2024-01-16T04:48:28.022 INFO:teuthology.orchestra.run.smithi156.stdout:(3/6): ceph-prometheus-alerts-16.2.14-449.ga70a 48 kB/s | 15 kB 00:00 2024-01-16T04:48:28.281 INFO:teuthology.orchestra.run.smithi156.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 115 kB/s | 34 kB 00:00 2024-01-16T04:48:28.406 INFO:teuthology.orchestra.run.smithi156.stdout:(5/6): ceph-mgr-dashboard-16.2.14-449.ga70aebef 4.5 MB/s | 3.1 MB 00:00 2024-01-16T04:48:28.440 INFO:teuthology.orchestra.run.smithi156.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 471 kB/s | 196 kB 00:00 2024-01-16T04:48:28.442 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:48:28.442 INFO:teuthology.orchestra.run.smithi156.stdout:Total 2.9 MB/s | 3.4 MB 00:01 2024-01-16T04:48:28.454 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:48:28.465 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:48:28.465 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:48:28.618 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:48:28.621 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:48:28.916 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:48:29.093 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-16T04:48:29.321 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-16T04:48:29.490 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-16T04:48:29.652 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-449.ga70aebef.el8.n 4/6 2024-01-16T04:48:30.646 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-449.ga70aebef.el8. 5/6 2024-01-16T04:48:30.905 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-449.ga70aebef.el8.noarc 6/6 2024-01-16T04:48:31.187 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-449.ga70aebef.el8.noarc 6/6 2024-01-16T04:48:31.188 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-449.ga70aebef.el8. 1/6 2024-01-16T04:48:31.188 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-449.ga70aebef.el8.noarc 2/6 2024-01-16T04:48:31.188 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-449.ga70aebef.el8.n 3/6 2024-01-16T04:48:31.188 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-16T04:48:31.188 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-16T04:48:31.917 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-16T04:48:31.917 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:31.917 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:48:31.917 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:48:31.917 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:48:31.917 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:48:31.917 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-16T04:48:31.917 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-16T04:48:31.917 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-16T04:48:31.918 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:31.918 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:48:32.040 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-16T04:48:32.901 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:41 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:48:34.622 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:48:34.624 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-01-16T04:48:34.624 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-01-16T04:48:34.624 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-01-16T04:48:34.624 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:48:34.624 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 8.6 M 2024-01-16T04:48:34.624 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-01-16T04:48:34.624 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-16T04:48:34.624 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-16T04:48:34.624 INFO:teuthology.orchestra.run.smithi156.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-16T04:48:34.624 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-16T04:48:34.624 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-16T04:48:34.624 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-16T04:48:34.624 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-16T04:48:34.625 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:34.625 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:48:34.625 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-01-16T04:48:34.625 INFO:teuthology.orchestra.run.smithi156.stdout:Install 8 Packages 2024-01-16T04:48:34.625 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:34.625 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 37 M 2024-01-16T04:48:34.625 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 210 M 2024-01-16T04:48:34.625 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:48:35.770 INFO:teuthology.orchestra.run.smithi156.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.14-44 9.4 MB/s | 8.6 MB 00:00 2024-01-16T04:48:36.487 INFO:teuthology.orchestra.run.smithi156.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 3.0 MB/s | 4.9 MB 00:01 2024-01-16T04:48:36.646 INFO:teuthology.orchestra.run.smithi156.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.4 MB/s | 225 kB 00:00 2024-01-16T04:48:37.755 INFO:teuthology.orchestra.run.smithi156.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 1.6 MB/s | 4.7 MB 00:02 2024-01-16T04:48:37.898 INFO:teuthology.orchestra.run.smithi156.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 4.5 MB/s | 645 kB 00:00 2024-01-16T04:48:37.940 INFO:teuthology.orchestra.run.smithi156.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 4.0 MB/s | 172 kB 00:00 2024-01-16T04:48:38.123 INFO:teuthology.orchestra.run.smithi156.stdout:(7/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 1.6 MB/s | 3.7 MB 00:02 2024-01-16T04:48:41.206 INFO:teuthology.orchestra.run.smithi156.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 3.1 MB/s | 14 MB 00:04 2024-01-16T04:48:41.207 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:48:41.208 INFO:teuthology.orchestra.run.smithi156.stdout:Total 5.6 MB/s | 37 MB 00:06 2024-01-16T04:48:41.444 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:48:41.463 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:48:41.463 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:48:41.814 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:48:41.817 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:48:42.368 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:48:42.494 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-16T04:48:42.661 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-16T04:48:42.835 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-16T04:48:43.431 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-16T04:48:43.627 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-16T04:48:44.244 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-16T04:48:44.385 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-16T04:48:45.076 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-16T04:48:45.397 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-16T04:48:45.652 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-16T04:48:47.421 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-16T04:48:48.687 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-16T04:48:48.884 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-449.ga70aebe 8/8 2024-01-16T04:48:49.230 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-449.ga70aebe 8/8 2024-01-16T04:48:49.230 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-449.ga70aebe 1/8 2024-01-16T04:48:49.230 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-16T04:48:49.230 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-16T04:48:49.231 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-16T04:48:49.231 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-16T04:48:49.231 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-16T04:48:49.231 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-16T04:48:49.959 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-16T04:48:49.959 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:49.959 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:48:49.960 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:48:49.960 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-16T04:48:49.960 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-16T04:48:49.960 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-16T04:48:49.960 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-16T04:48:49.960 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-16T04:48:49.960 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-16T04:48:49.961 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-16T04:48:49.961 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:49.961 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:48:50.083 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-rook 2024-01-16T04:48:50.934 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:01:59 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:48:52.650 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:48:52.652 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:48:52.652 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-01-16T04:48:52.652 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:48:52.652 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:48:52.652 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 27 k 2024-01-16T04:48:52.652 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-01-16T04:48:52.652 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-16T04:48:52.652 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-16T04:48:52.652 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-16T04:48:52.652 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-16T04:48:52.652 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-16T04:48:52.652 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-16T04:48:52.653 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-16T04:48:52.653 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-16T04:48:52.653 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-16T04:48:52.653 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-16T04:48:52.653 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-16T04:48:52.653 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-16T04:48:52.653 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:52.653 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:48:52.653 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:48:52.653 INFO:teuthology.orchestra.run.smithi156.stdout:Install 13 Packages 2024-01-16T04:48:52.653 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:52.654 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 1.9 M 2024-01-16T04:48:52.654 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 24 M 2024-01-16T04:48:52.654 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:48:53.197 INFO:teuthology.orchestra.run.smithi156.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 180 kB/s | 20 kB 00:00 2024-01-16T04:48:53.222 INFO:teuthology.orchestra.run.smithi156.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 195 kB/s | 27 kB 00:00 2024-01-16T04:48:53.297 INFO:teuthology.orchestra.run.smithi156.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2024-01-16T04:48:53.322 INFO:teuthology.orchestra.run.smithi156.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.1 MB/s | 110 kB 00:00 2024-01-16T04:48:53.364 INFO:teuthology.orchestra.run.smithi156.stdout:(5/13): ceph-mgr-rook-16.2.14-449.ga70aebef.el8 98 kB/s | 27 kB 00:00 2024-01-16T04:48:53.423 INFO:teuthology.orchestra.run.smithi156.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 424 kB/s | 43 kB 00:00 2024-01-16T04:48:53.448 INFO:teuthology.orchestra.run.smithi156.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.0 MB/s | 155 kB 00:00 2024-01-16T04:48:53.749 INFO:teuthology.orchestra.run.smithi156.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 86 kB/s | 33 kB 00:00 2024-01-16T04:48:53.766 INFO:teuthology.orchestra.run.smithi156.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 48 kB/s | 16 kB 00:00 2024-01-16T04:48:53.933 INFO:teuthology.orchestra.run.smithi156.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 169 kB/s | 81 kB 00:00 2024-01-16T04:48:54.000 INFO:teuthology.orchestra.run.smithi156.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 275 kB/s | 64 kB 00:00 2024-01-16T04:48:54.075 INFO:teuthology.orchestra.run.smithi156.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 432 kB/s | 61 kB 00:00 2024-01-16T04:48:54.275 INFO:teuthology.orchestra.run.smithi156.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.2 MB/s | 1.2 MB 00:00 2024-01-16T04:48:54.277 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:48:54.278 INFO:teuthology.orchestra.run.smithi156.stdout:Total 1.2 MB/s | 1.9 MB 00:01 2024-01-16T04:48:54.337 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:48:54.353 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:48:54.353 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:48:54.441 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:48:54.444 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:48:54.949 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:48:55.211 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-16T04:48:55.399 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-16T04:48:55.610 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-16T04:48:55.873 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-16T04:48:56.072 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-16T04:48:56.337 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-16T04:48:56.599 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-16T04:48:56.854 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-16T04:48:57.863 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-16T04:48:58.148 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-16T04:48:58.349 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-16T04:48:58.499 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-16T04:48:58.683 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-rook-2:16.2.14-449.ga70aebef.el8.noarch 13/13 2024-01-16T04:48:58.982 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-449.ga70aebef.el8.noarch 13/13 2024-01-16T04:48:58.983 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-rook-2:16.2.14-449.ga70aebef.el8.noarch 1/13 2024-01-16T04:48:58.983 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-16T04:48:58.983 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-16T04:48:58.983 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-16T04:48:58.983 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-16T04:48:58.983 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-16T04:48:58.983 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-16T04:48:58.983 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-16T04:48:58.983 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-16T04:48:58.983 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-16T04:48:58.983 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-16T04:48:58.983 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-16T04:48:59.772 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-16T04:48:59.772 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:59.772 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:48:59.772 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:48:59.772 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-16T04:48:59.772 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-16T04:48:59.772 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-16T04:48:59.772 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-16T04:48:59.773 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-16T04:48:59.773 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-16T04:48:59.773 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-16T04:48:59.773 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-16T04:48:59.773 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-16T04:48:59.773 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-16T04:48:59.773 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-16T04:48:59.773 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-16T04:48:59.773 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:48:59.773 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:48:59.896 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-mgr-cephadm 2024-01-16T04:49:00.745 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:09 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:49:02.456 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:49:02.457 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:02.457 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-01-16T04:49:02.457 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:02.458 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:49:02.458 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm noarch 2:16.2.14-449.ga70aebef.el8 ceph-noarch 116 k 2024-01-16T04:49:02.458 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-01-16T04:49:02.458 INFO:teuthology.orchestra.run.smithi156.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-16T04:49:02.458 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-16T04:49:02.458 INFO:teuthology.orchestra.run.smithi156.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-16T04:49:02.458 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-16T04:49:02.458 INFO:teuthology.orchestra.run.smithi156.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-16T04:49:02.458 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:02.458 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:49:02.458 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:02.459 INFO:teuthology.orchestra.run.smithi156.stdout:Install 6 Packages 2024-01-16T04:49:02.459 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:02.460 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 5.8 M 2024-01-16T04:49:02.460 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 24 M 2024-01-16T04:49:02.460 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:49:03.944 INFO:teuthology.orchestra.run.smithi156.stdout:(1/6): ceph-mgr-cephadm-16.2.14-449.ga70aebef.e 626 kB/s | 116 kB 00:00 2024-01-16T04:49:04.044 INFO:teuthology.orchestra.run.smithi156.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 212 kB/s | 21 kB 00:00 2024-01-16T04:49:04.086 INFO:teuthology.orchestra.run.smithi156.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.6 MB/s | 538 kB 00:00 2024-01-16T04:49:04.153 INFO:teuthology.orchestra.run.smithi156.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 3.2 MB/s | 355 kB 00:00 2024-01-16T04:49:04.187 INFO:teuthology.orchestra.run.smithi156.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 487 kB/s | 49 kB 00:00 2024-01-16T04:49:04.412 INFO:teuthology.orchestra.run.smithi156.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 7.3 MB/s | 4.8 MB 00:00 2024-01-16T04:49:04.413 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:49:04.413 INFO:teuthology.orchestra.run.smithi156.stdout:Total 3.0 MB/s | 5.8 MB 00:01 2024-01-16T04:49:04.467 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:49:04.478 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:49:04.478 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:49:04.584 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:49:04.587 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:49:04.938 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:49:05.214 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-16T04:49:05.442 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-16T04:49:06.389 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-16T04:49:06.743 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-16T04:49:06.981 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-16T04:49:07.190 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-449.ga70aebef.el8.noarch 6/6 2024-01-16T04:49:07.461 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-449.ga70aebef.el8.noarch 6/6 2024-01-16T04:49:07.461 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-449.ga70aebef.el8.noarch 1/6 2024-01-16T04:49:07.462 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-16T04:49:07.462 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-16T04:49:07.462 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-16T04:49:07.462 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-16T04:49:08.193 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-16T04:49:08.193 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:08.194 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:49:08.194 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm-2:16.2.14-449.ga70aebef.el8.noarch 2024-01-16T04:49:08.194 INFO:teuthology.orchestra.run.smithi156.stdout: python3-apipkg-1.5-6.el8.noarch 2024-01-16T04:49:08.194 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel-2.5.1-7.el8.noarch 2024-01-16T04:49:08.194 INFO:teuthology.orchestra.run.smithi156.stdout: python3-execnet-1.7.1-1.el8.noarch 2024-01-16T04:49:08.194 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-01-16T04:49:08.194 INFO:teuthology.orchestra.run.smithi156.stdout: python3-remoto-1.2.1-1.el8.noarch 2024-01-16T04:49:08.194 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:08.194 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:49:08.299 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-fuse 2024-01-16T04:49:09.158 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:18 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:49:10.897 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:49:10.898 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:10.898 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-01-16T04:49:10.898 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:10.898 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:49:10.898 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse x86_64 2:16.2.14-449.ga70aebef.el8 ceph 911 k 2024-01-16T04:49:10.898 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-01-16T04:49:10.898 INFO:teuthology.orchestra.run.smithi156.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-16T04:49:10.898 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:10.898 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:49:10.899 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:10.899 INFO:teuthology.orchestra.run.smithi156.stdout:Install 2 Packages 2024-01-16T04:49:10.899 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:10.899 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 994 k 2024-01-16T04:49:10.899 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 2.8 M 2024-01-16T04:49:10.899 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:49:11.312 INFO:teuthology.orchestra.run.smithi156.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 916 kB/s | 83 kB 00:00 2024-01-16T04:49:11.487 INFO:teuthology.orchestra.run.smithi156.stdout:(2/2): ceph-fuse-16.2.14-449.ga70aebef.el8.x86_ 3.3 MB/s | 911 kB 00:00 2024-01-16T04:49:11.488 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:49:11.488 INFO:teuthology.orchestra.run.smithi156.stdout:Total 1.6 MB/s | 994 kB 00:00 2024-01-16T04:49:11.498 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:49:11.518 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:49:11.519 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:49:11.605 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:49:11.608 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:49:11.783 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:49:12.092 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-16T04:49:12.305 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 2/2 2024-01-16T04:49:12.732 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 2/2 2024-01-16T04:49:12.732 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 1/2 2024-01-16T04:49:13.505 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-16T04:49:13.505 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:13.505 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:49:13.506 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-16T04:49:13.506 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:13.506 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:49:14.501 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install librados-devel 2024-01-16T04:49:15.373 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:24 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:49:17.104 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:49:17.105 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:17.105 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-01-16T04:49:17.105 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:17.105 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:49:17.105 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel x86_64 2:16.2.14-449.ga70aebef.el8 ceph 131 k 2024-01-16T04:49:17.105 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:17.105 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:49:17.105 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:17.106 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-01-16T04:49:17.106 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:17.106 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 131 k 2024-01-16T04:49:17.106 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 432 k 2024-01-16T04:49:17.106 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:49:17.292 INFO:teuthology.orchestra.run.smithi156.stdout:librados-devel-16.2.14-449.ga70aebef.el8.x86_64 705 kB/s | 131 kB 00:00 2024-01-16T04:49:17.293 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:49:17.293 INFO:teuthology.orchestra.run.smithi156.stdout:Total 700 kB/s | 131 kB 00:00 2024-01-16T04:49:17.293 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:49:17.298 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:49:17.298 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:49:17.347 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:49:17.351 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:49:17.481 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:49:17.695 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:49:17.990 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:49:18.679 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:49:18.679 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:18.679 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:49:18.679 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:49:18.679 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:18.679 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:49:18.871 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install libcephfs2 2024-01-16T04:49:19.745 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:28 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:49:21.229 INFO:teuthology.orchestra.run.smithi156.stdout:Package libcephfs2-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:49:21.467 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:49:21.468 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-01-16T04:49:21.468 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:49:21.569 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install libcephfs-devel 2024-01-16T04:49:22.429 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:31 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:49:24.145 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:49:24.145 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:24.145 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-01-16T04:49:24.145 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:24.145 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:49:24.145 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel x86_64 2:16.2.14-449.ga70aebef.el8 ceph 25 k 2024-01-16T04:49:24.145 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:24.146 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:49:24.146 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:24.146 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-01-16T04:49:24.146 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:24.146 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 25 k 2024-01-16T04:49:24.146 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 107 k 2024-01-16T04:49:24.147 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:49:24.366 INFO:teuthology.orchestra.run.smithi156.stdout:libcephfs-devel-16.2.14-449.ga70aebef.el8.x86_6 116 kB/s | 25 kB 00:00 2024-01-16T04:49:24.366 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:49:24.366 INFO:teuthology.orchestra.run.smithi156.stdout:Total 116 kB/s | 25 kB 00:00 2024-01-16T04:49:24.367 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:49:24.370 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:49:24.370 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:49:24.372 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:49:24.375 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:49:24.428 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:49:24.611 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:49:24.883 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:49:25.546 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs-devel-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:49:25.546 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:25.546 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:49:25.546 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:49:25.546 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:25.546 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:49:25.662 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install librados2 2024-01-16T04:49:26.527 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:35 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:49:28.009 INFO:teuthology.orchestra.run.smithi156.stdout:Package librados2-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:49:28.246 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:49:28.247 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-01-16T04:49:28.247 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:49:28.338 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install librbd1 2024-01-16T04:49:29.203 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:38 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:49:30.673 INFO:teuthology.orchestra.run.smithi156.stdout:Package librbd1-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:49:30.908 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:49:30.908 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-01-16T04:49:30.908 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:49:31.006 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install python3-rados 2024-01-16T04:49:31.866 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:40 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:49:33.335 INFO:teuthology.orchestra.run.smithi156.stdout:Package python3-rados-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:49:33.568 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:49:33.569 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-01-16T04:49:33.570 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:49:33.668 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install python3-rgw 2024-01-16T04:49:34.526 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:43 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:49:36.002 INFO:teuthology.orchestra.run.smithi156.stdout:Package python3-rgw-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:49:36.264 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:49:36.265 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-01-16T04:49:36.265 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:49:36.358 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install python3-cephfs 2024-01-16T04:49:37.217 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:46 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:49:38.678 INFO:teuthology.orchestra.run.smithi156.stdout:Package python3-cephfs-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:49:38.912 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:49:38.914 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-01-16T04:49:38.914 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:49:39.006 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install python3-rbd 2024-01-16T04:49:39.871 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:48 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:49:41.347 INFO:teuthology.orchestra.run.smithi156.stdout:Package python3-rbd-2:16.2.14-449.ga70aebef.el8.x86_64 is already installed. 2024-01-16T04:49:41.581 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:49:41.583 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-01-16T04:49:41.583 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:49:41.683 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install rbd-fuse 2024-01-16T04:49:42.544 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:51 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:49:44.262 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:49:44.262 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:44.262 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2024-01-16T04:49:44.262 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:44.263 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:49:44.263 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse x86_64 2:16.2.14-449.ga70aebef.el8 ceph 88 k 2024-01-16T04:49:44.263 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:44.263 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:49:44.263 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:44.263 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-01-16T04:49:44.263 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:44.263 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 88 k 2024-01-16T04:49:44.264 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 240 k 2024-01-16T04:49:44.264 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:49:44.431 INFO:teuthology.orchestra.run.smithi156.stdout:rbd-fuse-16.2.14-449.ga70aebef.el8.x86_64.rpm 529 kB/s | 88 kB 00:00 2024-01-16T04:49:44.431 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:49:44.431 INFO:teuthology.orchestra.run.smithi156.stdout:Total 525 kB/s | 88 kB 00:00 2024-01-16T04:49:44.432 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:49:44.441 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:49:44.441 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:49:44.491 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:49:44.494 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:49:44.613 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:49:44.799 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:49:45.106 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:49:45.807 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:49:45.808 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:45.808 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:49:45.808 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:49:45.808 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:45.808 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:49:46.007 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install rbd-mirror 2024-01-16T04:49:46.871 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:55 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:49:48.561 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:49:48.561 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:48.561 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repo Size 2024-01-16T04:49:48.561 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:48.561 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:49:48.561 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror x86_64 2:16.2.14-449.ga70aebef.el8 ceph 3.7 M 2024-01-16T04:49:48.562 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:48.562 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:49:48.562 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:48.562 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-01-16T04:49:48.562 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:48.562 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 3.7 M 2024-01-16T04:49:48.562 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 15 M 2024-01-16T04:49:48.562 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:49:49.176 INFO:teuthology.orchestra.run.smithi156.stdout:rbd-mirror-16.2.14-449.ga70aebef.el8.x86_64.rpm 6.0 MB/s | 3.7 MB 00:00 2024-01-16T04:49:49.177 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:49:49.177 INFO:teuthology.orchestra.run.smithi156.stdout:Total 6.0 MB/s | 3.7 MB 00:00 2024-01-16T04:49:49.177 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:49:49.189 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:49:49.189 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:49:49.252 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:49:49.254 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:49:49.704 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:49:49.950 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-mirror-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:49:50.375 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-mirror-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:49:51.102 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-mirror-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:49:51.102 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:51.102 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:49:51.102 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:49:51.102 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:51.102 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:49:51.308 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install rbd-nbd 2024-01-16T04:49:52.177 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:03:01 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:49:53.895 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:49:53.896 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:53.896 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2024-01-16T04:49:53.896 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:53.896 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:49:53.896 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd x86_64 2:16.2.14-449.ga70aebef.el8 ceph 175 k 2024-01-16T04:49:53.896 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:53.896 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:49:53.897 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:49:53.897 INFO:teuthology.orchestra.run.smithi156.stdout:Install 1 Package 2024-01-16T04:49:53.897 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:53.897 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 175 k 2024-01-16T04:49:53.897 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 523 k 2024-01-16T04:49:53.897 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:49:54.096 INFO:teuthology.orchestra.run.smithi156.stdout:rbd-nbd-16.2.14-449.ga70aebef.el8.x86_64.rpm 878 kB/s | 175 kB 00:00 2024-01-16T04:49:54.097 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:49:54.097 INFO:teuthology.orchestra.run.smithi156.stdout:Total 872 kB/s | 175 kB 00:00 2024-01-16T04:49:54.097 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:49:54.108 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:49:54.108 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:49:54.158 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:49:54.161 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:49:54.294 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:49:54.481 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-nbd-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:49:54.770 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-nbd-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:49:55.470 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-nbd-2:16.2.14-449.ga70aebef.el8.x86_64 1/1 2024-01-16T04:49:55.470 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:55.470 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:49:55.470 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-2:16.2.14-449.ga70aebef.el8.x86_64 2024-01-16T04:49:55.470 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:55.470 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:49:55.694 DEBUG:teuthology.parallel:result is None 2024-01-16T04:49:55.694 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a70aebef70df6ec2d957347ccbfc484e0573dd5f 2024-01-16T04:49:55.951 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-16T04:49:55.973 INFO:teuthology.orchestra.run.smithi156.stdout:16.2.14-449.ga70aebef.el8 2024-01-16T04:49:55.974 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-449.ga70aebef.el8 2024-01-16T04:49:55.974 INFO:teuthology.task.install:The correct ceph version 16.2.14-449.ga70aebef is installed. 2024-01-16T04:49:55.975 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-16T04:49:55.976 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:49:55.976 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-16T04:49:56.041 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-16T04:49:56.041 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:49:56.041 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/daemon-helper 2024-01-16T04:49:56.108 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-16T04:49:56.174 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-16T04:49:56.174 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:49:56.174 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-16T04:49:56.241 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-16T04:49:56.311 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-16T04:49:56.311 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:49:56.311 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/stdin-killer 2024-01-16T04:49:56.378 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-16T04:49:56.446 INFO:teuthology.run_tasks:Running task ceph... 2024-01-16T04:49:56.520 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-16T04:49:56.520 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /var/log/ceph 2024-01-16T04:49:56.547 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-16T04:49:56.547 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-16T04:49:56.614 INFO:tasks.ceph:Creating extra log directories... 2024-01-16T04:49:56.614 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-16T04:49:56.689 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-16T04:49:56.689 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/transit/', 'rgw crypt vault secret engine': 'transit', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-16T04:49:56.689 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517339', 'branch': 'wip-yuri5-testing-2024-01-11-1300-pacific', 'description': 'rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7517339', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri5-testing-2024-01-11-1300-pacific'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/transit/', 'rgw crypt vault secret engine': 'transit', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f'}}, 'rgw': {'client.0': {'use-vault-role': 'client.0'}}, 'workunit': {'branch': 'wip-yuri5-testing-2024-01-11-1300-pacific', 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9013, 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-yuri5-testing-2024-01-11-1300-pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f', 'targets': {'smithi156.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNs8qpHHPPUS2s7++BnIXXAgH3hvybrKv4GM3m7Zbb48nI7wNNZl355X3XAuPc/Bw79V8QAbjDfp0cJPFstEVA4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'vault': {'client.0': {'engine': 'transit', 'install_sha256': '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458', 'install_url': 'https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip', 'prefix': '/v1/transit/keys/', 'root_token': 'test_root_token', 'secrets': [{'path': 'my-key-1'}, {'path': 'my-key-2'}]}}}, {'rgw': {'client.0': None}}, {'s3tests': {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-pacific', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-15_22:05:22', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-16T04:49:56.689 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-16T04:49:56.739 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/run/ceph 2024-01-16T04:49:56.806 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:49:56.806 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2024-01-16T04:49:56.865 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-16T04:49:56.866 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2024-01-16T04:49:56.922 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-16T04:49:56.923 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T04:49:56.923 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 163187 Links: 1 2024-01-16T04:49:56.923 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T04:49:56.923 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T04:49:56.923 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-01-16 04:49:50.123910670 +0000 2024-01-16T04:49:56.923 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-01-16 04:45:42.302879087 +0000 2024-01-16T04:49:56.923 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-01-16 04:45:42.302879087 +0000 2024-01-16T04:49:56.923 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-01-16T04:49:56.923 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-16T04:49:56.987 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-01-16T04:49:56.987 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-01-16T04:49:56.987 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 7.7863e-05 s, 6.6 MB/s 2024-01-16T04:49:56.988 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-16T04:49:57.047 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2024-01-16T04:49:57.104 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-16T04:49:57.105 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T04:49:57.105 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 163293 Links: 1 2024-01-16T04:49:57.105 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T04:49:57.105 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T04:49:57.105 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-01-16 04:49:50.123910670 +0000 2024-01-16T04:49:57.105 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-01-16 04:45:42.643874102 +0000 2024-01-16T04:49:57.105 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-01-16 04:45:42.643874102 +0000 2024-01-16T04:49:57.105 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-01-16T04:49:57.105 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-16T04:49:57.170 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-01-16T04:49:57.171 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-01-16T04:49:57.171 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000123937 s, 4.1 MB/s 2024-01-16T04:49:57.172 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-16T04:49:57.230 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2024-01-16T04:49:57.288 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-16T04:49:57.288 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T04:49:57.288 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 157440 Links: 1 2024-01-16T04:49:57.289 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T04:49:57.289 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T04:49:57.289 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-01-16 04:49:50.123910670 +0000 2024-01-16T04:49:57.289 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-01-16 04:45:42.990869030 +0000 2024-01-16T04:49:57.289 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-01-16 04:45:42.990869030 +0000 2024-01-16T04:49:57.289 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-01-16T04:49:57.289 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-16T04:49:57.354 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-01-16T04:49:57.354 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-01-16T04:49:57.354 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000120001 s, 4.3 MB/s 2024-01-16T04:49:57.355 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-16T04:49:57.414 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2024-01-16T04:49:57.472 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-16T04:49:57.472 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T04:49:57.472 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 165902 Links: 1 2024-01-16T04:49:57.472 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T04:49:57.472 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T04:49:57.472 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-01-16 04:49:50.123910670 +0000 2024-01-16T04:49:57.472 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-01-16 04:45:43.328864089 +0000 2024-01-16T04:49:57.472 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-01-16 04:45:43.328864089 +0000 2024-01-16T04:49:57.472 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2024-01-16T04:49:57.473 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-16T04:49:57.537 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-01-16T04:49:57.537 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-01-16T04:49:57.538 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 8.3011e-05 s, 6.2 MB/s 2024-01-16T04:49:57.538 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-16T04:49:57.597 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-16T04:49:57.597 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi156.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-16T04:49:57.597 INFO:tasks.ceph:Generating config... 2024-01-16T04:49:57.598 INFO:tasks.ceph:[client] debug rgw = 20 2024-01-16T04:49:57.598 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2024-01-16T04:49:57.598 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = vault 2024-01-16T04:49:57.598 INFO:tasks.ceph:[client] rgw crypt vault auth = token 2024-01-16T04:49:57.598 INFO:tasks.ceph:[client] rgw crypt vault prefix = /v1/transit/ 2024-01-16T04:49:57.598 INFO:tasks.ceph:[client] rgw crypt vault secret engine = transit 2024-01-16T04:49:57.598 INFO:tasks.ceph:[client] setgroup = ceph 2024-01-16T04:49:57.598 INFO:tasks.ceph:[client] setuser = ceph 2024-01-16T04:49:57.599 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-16T04:49:57.599 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-16T04:49:57.599 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-16T04:49:57.599 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-16T04:49:57.599 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-16T04:49:57.599 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-16T04:49:57.599 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-16T04:49:57.599 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-01-16T04:49:57.599 INFO:tasks.ceph:Setting up mon.a... 2024-01-16T04:49:57.599 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-16T04:49:57.737 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/ceph.keyring 2024-01-16T04:49:57.738 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-16T04:49:57.779 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T04:49:57.846 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.156')] 2024-01-16T04:49:57.847 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.156'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'vault', 'rgw crypt vault auth': 'token', 'rgw crypt vault prefix': '/v1/transit/', 'rgw crypt vault secret engine': 'transit', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2024-01-16T04:49:57.847 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:49:57.847 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-16T04:49:57.904 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.156 --print /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:49:57.981 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:49:57.981 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:49:57.982 INFO:teuthology.orchestra.run.smithi156.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:49:57.982 INFO:teuthology.orchestra.run.smithi156.stdout:monmaptool: generated fsid dae07de8-a7c7-4fa7-b3ca-19c915fbdf1d 2024-01-16T04:49:57.982 INFO:teuthology.orchestra.run.smithi156.stdout:epoch 0 2024-01-16T04:49:57.982 INFO:teuthology.orchestra.run.smithi156.stdout:fsid dae07de8-a7c7-4fa7-b3ca-19c915fbdf1d 2024-01-16T04:49:57.982 INFO:teuthology.orchestra.run.smithi156.stdout:last_changed 2024-01-16T04:49:57.985983+0000 2024-01-16T04:49:57.982 INFO:teuthology.orchestra.run.smithi156.stdout:created 2024-01-16T04:49:57.985983+0000 2024-01-16T04:49:57.982 INFO:teuthology.orchestra.run.smithi156.stdout:min_mon_release 0 (unknown) 2024-01-16T04:49:57.983 INFO:teuthology.orchestra.run.smithi156.stdout:election_strategy: 1 2024-01-16T04:49:57.983 INFO:teuthology.orchestra.run.smithi156.stdout:0: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.a 2024-01-16T04:49:57.983 INFO:teuthology.orchestra.run.smithi156.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-01-16T04:49:57.984 DEBUG:teuthology.orchestra.run.smithi156:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-16T04:49:58.042 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID dae07de8-a7c7-4fa7-b3ca-19c915fbdf1d... 2024-01-16T04:49:58.043 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-16T04:49:58.127 INFO:teuthology.orchestra.run.smithi156.stdout:[global] 2024-01-16T04:49:58.127 INFO:teuthology.orchestra.run.smithi156.stdout: chdir = "" 2024-01-16T04:49:58.127 INFO:teuthology.orchestra.run.smithi156.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-16T04:49:58.127 INFO:teuthology.orchestra.run.smithi156.stdout: auth supported = cephx 2024-01-16T04:49:58.127 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.127 INFO:teuthology.orchestra.run.smithi156.stdout: filestore xattr use omap = true 2024-01-16T04:49:58.127 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.127 INFO:teuthology.orchestra.run.smithi156.stdout: mon clock drift allowed = 1.000 2024-01-16T04:49:58.127 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.127 INFO:teuthology.orchestra.run.smithi156.stdout: osd crush chooseleaf type = 0 2024-01-16T04:49:58.127 INFO:teuthology.orchestra.run.smithi156.stdout: auth debug = true 2024-01-16T04:49:58.127 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.128 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on old message = true 2024-01-16T04:49:58.128 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on bug = true 2024-01-16T04:49:58.128 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.128 INFO:teuthology.orchestra.run.smithi156.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-16T04:49:58.128 INFO:teuthology.orchestra.run.smithi156.stdout: mon pg warn max object skew = 0 2024-01-16T04:49:58.128 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.128 INFO:teuthology.orchestra.run.smithi156.stdout: # disable pg_autoscaler by default for new pools 2024-01-16T04:49:58.128 INFO:teuthology.orchestra.run.smithi156.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-16T04:49:58.128 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.128 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default size = 2 2024-01-16T04:49:58.128 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.128 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow primary affinity = true 2024-01-16T04:49:58.128 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow pg remap = true 2024-01-16T04:49:58.128 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on legacy crush tunables = false 2024-01-16T04:49:58.129 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on crush straw calc version zero = false 2024-01-16T04:49:58.129 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on no sortbitwise = false 2024-01-16T04:49:58.129 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on osd down out interval zero = false 2024-01-16T04:49:58.129 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on too few osds = false 2024-01-16T04:49:58.129 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-16T04:49:58.129 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-16T04:49:58.129 INFO:teuthology.orchestra.run.smithi156.stdout: mon_allow_pool_size_one = true 2024-01-16T04:49:58.129 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.129 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2024-01-16T04:49:58.129 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.129 INFO:teuthology.orchestra.run.smithi156.stdout: osd default data pool replay window = 5 2024-01-16T04:49:58.129 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.129 INFO:teuthology.orchestra.run.smithi156.stdout: mon allow pool delete = true 2024-01-16T04:49:58.129 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.129 INFO:teuthology.orchestra.run.smithi156.stdout: mon cluster log file level = debug 2024-01-16T04:49:58.129 INFO:teuthology.orchestra.run.smithi156.stdout: debug asserts on shutdown = true 2024-01-16T04:49:58.130 INFO:teuthology.orchestra.run.smithi156.stdout: mon health detail to clog = false 2024-01-16T04:49:58.130 INFO:teuthology.orchestra.run.smithi156.stdout: mon host = 172.21.15.156 2024-01-16T04:49:58.130 INFO:teuthology.orchestra.run.smithi156.stdout: fsid = dae07de8-a7c7-4fa7-b3ca-19c915fbdf1d 2024-01-16T04:49:58.130 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.130 INFO:teuthology.orchestra.run.smithi156.stdout:[osd] 2024-01-16T04:49:58.130 INFO:teuthology.orchestra.run.smithi156.stdout: osd journal size = 100 2024-01-16T04:49:58.130 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.130 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub load threshold = 5.0 2024-01-16T04:49:58.130 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub max interval = 600 2024-01-16T04:49:58.130 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.130 INFO:teuthology.orchestra.run.smithi156.stdout: osd recover clone overlap = true 2024-01-16T04:49:58.130 INFO:teuthology.orchestra.run.smithi156.stdout: osd recovery max chunk = 1048576 2024-01-16T04:49:58.130 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.130 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug shutdown = true 2024-01-16T04:49:58.130 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug op order = true 2024-01-16T04:49:58.131 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug verify stray on activate = true 2024-01-16T04:49:58.131 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.131 INFO:teuthology.orchestra.run.smithi156.stdout: osd open classes on start = true 2024-01-16T04:49:58.131 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug pg log writeout = true 2024-01-16T04:49:58.131 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.131 INFO:teuthology.orchestra.run.smithi156.stdout: osd deep scrub update digest min age = 30 2024-01-16T04:49:58.131 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.131 INFO:teuthology.orchestra.run.smithi156.stdout: osd map max advance = 10 2024-01-16T04:49:58.131 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.131 INFO:teuthology.orchestra.run.smithi156.stdout: journal zero on create = true 2024-01-16T04:49:58.131 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.131 INFO:teuthology.orchestra.run.smithi156.stdout: filestore ondisk finisher threads = 3 2024-01-16T04:49:58.131 INFO:teuthology.orchestra.run.smithi156.stdout: filestore apply finisher threads = 3 2024-01-16T04:49:58.131 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.131 INFO:teuthology.orchestra.run.smithi156.stdout: bdev debug aio = true 2024-01-16T04:49:58.131 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug misdirected ops = true 2024-01-16T04:49:58.132 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-01-16T04:49:58.132 INFO:teuthology.orchestra.run.smithi156.stdout: debug osd = 20 2024-01-16T04:49:58.132 INFO:teuthology.orchestra.run.smithi156.stdout: osd shutdown pgref assert = True 2024-01-16T04:49:58.132 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.132 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr] 2024-01-16T04:49:58.132 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-01-16T04:49:58.132 INFO:teuthology.orchestra.run.smithi156.stdout: debug mgr = 20 2024-01-16T04:49:58.132 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2024-01-16T04:49:58.132 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2024-01-16T04:49:58.132 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2024-01-16T04:49:58.132 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2024-01-16T04:49:58.132 INFO:teuthology.orchestra.run.smithi156.stdout: mgr/telemetry/nag = false 2024-01-16T04:49:58.132 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.132 INFO:teuthology.orchestra.run.smithi156.stdout:[mon] 2024-01-16T04:49:58.132 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-01-16T04:49:58.133 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2024-01-16T04:49:58.133 INFO:teuthology.orchestra.run.smithi156.stdout: debug paxos = 20 2024-01-16T04:49:58.133 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2024-01-16T04:49:58.133 INFO:teuthology.orchestra.run.smithi156.stdout: mon data avail warn = 5 2024-01-16T04:49:58.133 INFO:teuthology.orchestra.run.smithi156.stdout: mon mgr mkfs grace = 240 2024-01-16T04:49:58.133 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2024-01-16T04:49:58.133 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd reporter subtree level = osd 2024-01-16T04:49:58.133 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd prime pg temp = true 2024-01-16T04:49:58.133 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2024-01-16T04:49:58.133 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.133 INFO:teuthology.orchestra.run.smithi156.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-16T04:49:58.133 INFO:teuthology.orchestra.run.smithi156.stdout: auth mon ticket ttl = 660 # 11m 2024-01-16T04:49:58.133 INFO:teuthology.orchestra.run.smithi156.stdout: auth service ticket ttl = 240 # 4m 2024-01-16T04:49:58.133 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.133 INFO:teuthology.orchestra.run.smithi156.stdout: # don't complain about insecure global_id in the test suite 2024-01-16T04:49:58.134 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-16T04:49:58.134 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-16T04:49:58.134 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:49:58.134 INFO:teuthology.orchestra.run.smithi156.stdout:[client] 2024-01-16T04:49:58.134 INFO:teuthology.orchestra.run.smithi156.stdout: rgw cache enabled = true 2024-01-16T04:49:58.134 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable ops log = true 2024-01-16T04:49:58.134 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable usage log = true 2024-01-16T04:49:58.134 INFO:teuthology.orchestra.run.smithi156.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-16T04:49:58.134 INFO:teuthology.orchestra.run.smithi156.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-16T04:49:58.134 INFO:teuthology.orchestra.run.smithi156.stdout: debug rgw = 20 2024-01-16T04:49:58.134 INFO:teuthology.orchestra.run.smithi156.stdout: rgw crypt require ssl = False 2024-01-16T04:49:58.134 INFO:teuthology.orchestra.run.smithi156.stdout: rgw crypt s3 kms backend = vault 2024-01-16T04:49:58.134 INFO:teuthology.orchestra.run.smithi156.stdout: rgw crypt vault auth = token 2024-01-16T04:49:58.134 INFO:teuthology.orchestra.run.smithi156.stdout: rgw crypt vault prefix = /v1/transit/ 2024-01-16T04:49:58.134 INFO:teuthology.orchestra.run.smithi156.stdout: rgw crypt vault secret engine = transit 2024-01-16T04:49:58.134 INFO:teuthology.orchestra.run.smithi156.stdout: setgroup = ceph 2024-01-16T04:49:58.135 INFO:teuthology.orchestra.run.smithi156.stdout: setuser = ceph 2024-01-16T04:49:58.135 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.a] 2024-01-16T04:49:58.137 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-16T04:49:58.138 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-16T04:49:58.229 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-16T04:49:58.229 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:49:58.229 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-16T04:49:58.247 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:49:58.247 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-16T04:49:58.304 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi156.front.sepia.ceph.com 2024-01-16T04:49:58.305 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:49:58.305 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-16T04:49:58.305 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T04:49:58.380 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:49:58.380 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:49:58.438 INFO:tasks.ceph:Setting up mon nodes... 2024-01-16T04:49:58.438 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-16T04:49:58.439 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-16T04:49:58.527 INFO:teuthology.orchestra.run.smithi156.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-16T04:49:58.529 INFO:tasks.ceph:Setting up mds nodes... 2024-01-16T04:49:58.529 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-16T04:49:58.529 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-16T04:49:58.568 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-16T04:49:58.580 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-16T04:49:58.580 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi156.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-16T04:49:58.580 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-16T04:49:58.647 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-16T04:49:58.647 INFO:tasks.ceph:role: osd.0 2024-01-16T04:49:58.647 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi156.front.sepia.ceph.com 2024-01-16T04:49:58.647 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-16T04:49:58.713 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T04:49:58.714 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T04:49:58.714 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T04:49:58.714 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T04:49:58.714 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T04:49:58.714 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-01-16T04:49:58.714 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T04:49:58.714 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T04:49:58.714 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T04:49:58.714 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T04:49:58.736 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-01-16T04:49:58.738 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-01-16T04:49:58.738 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-16T04:49:58.818 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-16T04:49:58.889 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-16T04:49:58.957 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-16T04:49:58.957 INFO:tasks.ceph:role: osd.1 2024-01-16T04:49:58.957 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi156.front.sepia.ceph.com 2024-01-16T04:49:58.957 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-16T04:49:59.023 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T04:49:59.023 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T04:49:59.023 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T04:49:59.023 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T04:49:59.023 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T04:49:59.023 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-01-16T04:49:59.023 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T04:49:59.023 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T04:49:59.023 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T04:49:59.023 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T04:49:59.045 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-01-16T04:49:59.047 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-01-16T04:49:59.047 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-16T04:49:59.126 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-16T04:49:59.197 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-16T04:49:59.264 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-16T04:49:59.264 INFO:tasks.ceph:role: osd.2 2024-01-16T04:49:59.264 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi156.front.sepia.ceph.com 2024-01-16T04:49:59.264 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-16T04:49:59.331 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T04:49:59.331 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T04:49:59.331 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T04:49:59.331 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-01-16T04:49:59.331 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T04:49:59.331 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-01-16T04:49:59.331 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T04:49:59.331 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-01-16T04:49:59.331 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T04:49:59.332 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T04:49:59.353 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-01-16T04:49:59.355 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-01-16T04:49:59.355 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-16T04:49:59.435 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-16T04:49:59.506 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:49:59.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:49:59.606+0000 7f97835a0380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-16T04:49:59.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:49:59.606+0000 7f97835a0380 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-16T04:49:59.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:49:59.606+0000 7f97835a0380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T04:49:59.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:49:59.606+0000 7f97835a0380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T04:49:59.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:49:59.606+0000 7f97835a0380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T04:49:59.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:49:59.607+0000 7f97835a0380 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-16T04:50:01.681 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-16T04:50:01.713 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:50:01.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:01.804+0000 7f16510da380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-16T04:50:01.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:01.804+0000 7f16510da380 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-16T04:50:01.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:01.804+0000 7f16510da380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T04:50:01.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:01.804+0000 7f16510da380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T04:50:01.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:01.804+0000 7f16510da380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T04:50:01.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:01.804+0000 7f16510da380 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-16T04:50:03.873 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-16T04:50:03.905 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:50:03.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:03.996+0000 7f0b36ac0380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-16T04:50:03.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:03.996+0000 7f0b36ac0380 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-16T04:50:03.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:03.996+0000 7f0b36ac0380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T04:50:03.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:03.996+0000 7f0b36ac0380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T04:50:03.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:03.996+0000 7f0b36ac0380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T04:50:03.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:03.997+0000 7f0b36ac0380 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-16T04:50:06.068 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-16T04:50:06.100 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-16T04:50:06.101 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:50:06.101 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-16T04:50:06.170 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:50:06.171 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-16T04:50:06.239 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:50:06.240 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-16T04:50:06.308 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:50:06.309 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-16T04:50:06.376 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:50:06.377 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-16T04:50:06.435 INFO:tasks.ceph:Adding keys to all mons... 2024-01-16T04:50:06.435 DEBUG:teuthology.orchestra.run.smithi156:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-16T04:50:06.501 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr.x] 2024-01-16T04:50:06.502 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQD2CqZlQoCmHxAAvQeguXIlvnnZgupX53KCMw== 2024-01-16T04:50:06.502 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.0] 2024-01-16T04:50:06.502 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQD3CqZlaso7JBAAdarIxPccNolYepyyQYiOdg== 2024-01-16T04:50:06.502 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.1] 2024-01-16T04:50:06.502 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQD5CqZl848EMBAA49Gdt0md0TIRE5l01SwA1g== 2024-01-16T04:50:06.502 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.2] 2024-01-16T04:50:06.502 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQD7CqZlOnt7OxAAzK+YFBPigEkQ1jlH7V6ItQ== 2024-01-16T04:50:06.502 INFO:teuthology.orchestra.run.smithi156.stdout:[client.0] 2024-01-16T04:50:06.502 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQD2CqZlVJMdIhAAPmE324iWYXydF2Q3QE4zaA== 2024-01-16T04:50:06.503 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T04:50:06.593 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T04:50:06.642 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T04:50:06.690 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T04:50:06.738 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T04:50:06.788 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-16T04:50:06.788 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-16T04:50:06.816 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T04:50:06.909 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-16T04:50:06.935 DEBUG:teuthology.orchestra.run.smithi156:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-16T04:50:06.993 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-16T04:50:06.993 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-16T04:50:06.993 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-16T04:50:07.036 INFO:tasks.ceph.mon.a:Started 2024-01-16T04:50:07.036 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-16T04:50:07.036 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-16T04:50:07.036 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-16T04:50:07.038 INFO:tasks.ceph.mgr.x:Started 2024-01-16T04:50:07.038 INFO:tasks.ceph:Setting crush tunables to default 2024-01-16T04:50:07.038 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd crush tunables default 2024-01-16T04:50:07.236 INFO:teuthology.orchestra.run.smithi156.stderr:adjusted tunables profile to default 2024-01-16T04:50:07.247 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-16T04:50:07.247 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:50:07.247 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-16T04:50:07.274 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:50:07.274 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-16T04:50:07.341 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:50:07.341 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-16T04:50:07.408 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 2fd312a6-3d18-421f-b641-a81c9bb49196 0 2024-01-16T04:50:07.462 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:07.465+0000 7fdd585e1680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-16T04:50:07.535 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:07.537+0000 7fdd585e1680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-16T04:50:07.653 INFO:teuthology.orchestra.run.smithi156.stdout:0 2024-01-16T04:50:07.666 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 9189293d-261d-4048-a9de-6a5778c6ce44 1 2024-01-16T04:50:07.908 INFO:teuthology.orchestra.run.smithi156.stdout:1 2024-01-16T04:50:07.921 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 65c3a83b-fea3-406d-a04b-67e1cdd6856e 2 2024-01-16T04:50:08.002 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:08.004+0000 7fdd585e1680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-16T04:50:08.133 INFO:teuthology.orchestra.run.smithi156.stdout:2 2024-01-16T04:50:08.145 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-16T04:50:08.145 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-01-16T04:50:08.148 INFO:tasks.ceph.osd.0:Started 2024-01-16T04:50:08.148 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-16T04:50:08.148 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-01-16T04:50:08.153 INFO:tasks.ceph.osd.1:Started 2024-01-16T04:50:08.153 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-16T04:50:08.153 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-01-16T04:50:08.155 INFO:tasks.ceph.osd.2:Started 2024-01-16T04:50:08.156 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T04:50:08.369 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:50:08.369 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":5,"fsid":"dae07de8-a7c7-4fa7-b3ca-19c915fbdf1d","created":"2024-01-16T04:50:07.150411+0000","modified":"2024-01-16T04:50:08.136013+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T04:50:08.136013+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"2fd312a6-3d18-421f-b641-a81c9bb49196","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9189293d-261d-4048-a9de-6a5778c6ce44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"65c3a83b-fea3-406d-a04b-67e1cdd6856e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:50:08.385 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-16T04:50:08.385 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-16T04:50:08.687 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T04:50:08.743 INFO:tasks.ceph.osd.0.smithi156.stderr:2024-01-16T04:50:08.746+0000 7ff1cac06380 -1 Falling back to public interface 2024-01-16T04:50:08.759 INFO:tasks.ceph.osd.1.smithi156.stderr:2024-01-16T04:50:08.762+0000 7f4ab22c4380 -1 Falling back to public interface 2024-01-16T04:50:08.788 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:08.790+0000 7fdd585e1680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-16T04:50:08.791 INFO:tasks.ceph.osd.2.smithi156.stderr:2024-01-16T04:50:08.793+0000 7fb12e404380 -1 Falling back to public interface 2024-01-16T04:50:08.859 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:08.861+0000 7fdd585e1680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-16T04:50:08.891 INFO:teuthology.misc.health.smithi156.stdout: 2024-01-16T04:50:08.891 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":5,"fsid":"dae07de8-a7c7-4fa7-b3ca-19c915fbdf1d","created":"2024-01-16T04:50:07.150411+0000","modified":"2024-01-16T04:50:08.136013+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T04:50:08.136013+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"2fd312a6-3d18-421f-b641-a81c9bb49196","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9189293d-261d-4048-a9de-6a5778c6ce44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"65c3a83b-fea3-406d-a04b-67e1cdd6856e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:50:08.903 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-01-16T04:50:09.050 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:09.053+0000 7fdd585e1680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-16T04:50:09.488 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:09.491+0000 7fdd585e1680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-16T04:50:09.591 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:09.594+0000 7fdd585e1680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-16T04:50:09.746 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:09.749+0000 7fdd585e1680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-16T04:50:09.826 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:09.826+0000 7fdd585e1680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-16T04:50:09.894 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:09.896+0000 7fdd585e1680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-16T04:50:10.163 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:10.155+0000 7fdd585e1680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-16T04:50:10.350 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:10.352+0000 7fdd585e1680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-16T04:50:11.025 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:11.027+0000 7fdd585e1680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-16T04:50:11.094 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:11.096+0000 7fdd585e1680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-16T04:50:11.174 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:11.176+0000 7fdd585e1680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-16T04:50:11.328 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:11.330+0000 7fdd585e1680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-16T04:50:11.398 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:11.400+0000 7fdd585e1680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-16T04:50:11.452 INFO:tasks.ceph.osd.0.smithi156.stderr:2024-01-16T04:50:11.454+0000 7ff1cac06380 -1 osd.0 0 log_to_monitors {default=true} 2024-01-16T04:50:11.501 INFO:tasks.ceph.osd.1.smithi156.stderr:2024-01-16T04:50:11.504+0000 7f4ab22c4380 -1 osd.1 0 log_to_monitors {default=true} 2024-01-16T04:50:11.520 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:11.523+0000 7fdd585e1680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-16T04:50:11.569 INFO:tasks.ceph.osd.2.smithi156.stderr:2024-01-16T04:50:11.572+0000 7fb12e404380 -1 osd.2 0 log_to_monitors {default=true} 2024-01-16T04:50:11.631 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:11.634+0000 7fdd585e1680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-16T04:50:11.952 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:11.955+0000 7fdd585e1680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-16T04:50:12.022 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:12.025+0000 7fdd585e1680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-16T04:50:12.511 INFO:tasks.ceph.osd.0.smithi156.stderr:2024-01-16T04:50:12.514+0000 7ff1c15a7700 -1 osd.0 0 waiting for initial osdmap 2024-01-16T04:50:12.512 INFO:tasks.ceph.osd.1.smithi156.stderr:2024-01-16T04:50:12.514+0000 7f4aa8c65700 -1 osd.1 0 waiting for initial osdmap 2024-01-16T04:50:12.513 INFO:tasks.ceph.osd.0.smithi156.stderr:2024-01-16T04:50:12.516+0000 7ff1bdd23700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T04:50:12.513 INFO:tasks.ceph.osd.1.smithi156.stderr:2024-01-16T04:50:12.516+0000 7f4aa53e1700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T04:50:13.042 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-01-16T04:50:13.044+0000 7fdd0dafa700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-01-16T04:50:13.516 INFO:tasks.ceph.osd.2.smithi156.stderr:2024-01-16T04:50:13.518+0000 7fb124da5700 -1 osd.2 0 waiting for initial osdmap 2024-01-16T04:50:13.550 INFO:tasks.ceph.osd.2.smithi156.stderr:2024-01-16T04:50:13.552+0000 7fb121521700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T04:50:15.206 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T04:50:15.497 INFO:teuthology.misc.health.smithi156.stdout: 2024-01-16T04:50:15.497 INFO:teuthology.misc.health.smithi156.stdout:{"epoch":9,"fsid":"dae07de8-a7c7-4fa7-b3ca-19c915fbdf1d","created":"2024-01-16T04:50:07.150411+0000","modified":"2024-01-16T04:50:14.516683+0000","last_up_change":"2024-01-16T04:50:14.516683+0000","last_in_change":"2024-01-16T04:50:08.136013+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-16T04:50:13.551691+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"2fd312a6-3d18-421f-b641-a81c9bb49196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":40342},{"type":"v1","addr":"172.21.15.156:6801","nonce":40342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":40342},{"type":"v1","addr":"172.21.15.156:6803","nonce":40342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":40342},{"type":"v1","addr":"172.21.15.156:6807","nonce":40342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":40342},{"type":"v1","addr":"172.21.15.156:6805","nonce":40342}]},"public_addr":"172.21.15.156:6801/40342","cluster_addr":"172.21.15.156:6803/40342","heartbeat_back_addr":"172.21.15.156:6807/40342","heartbeat_front_addr":"172.21.15.156:6805/40342","state":["exists","up"]},{"osd":1,"uuid":"9189293d-261d-4048-a9de-6a5778c6ce44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":40343},{"type":"v1","addr":"172.21.15.156:6809","nonce":40343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":40343},{"type":"v1","addr":"172.21.15.156:6811","nonce":40343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":40343},{"type":"v1","addr":"172.21.15.156:6815","nonce":40343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":40343},{"type":"v1","addr":"172.21.15.156:6813","nonce":40343}]},"public_addr":"172.21.15.156:6809/40343","cluster_addr":"172.21.15.156:6811/40343","heartbeat_back_addr":"172.21.15.156:6815/40343","heartbeat_front_addr":"172.21.15.156:6813/40343","state":["exists","up"]},{"osd":2,"uuid":"65c3a83b-fea3-406d-a04b-67e1cdd6856e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":40372},{"type":"v1","addr":"172.21.15.156:6817","nonce":40372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":40372},{"type":"v1","addr":"172.21.15.156:6819","nonce":40372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":40372},{"type":"v1","addr":"172.21.15.156:6823","nonce":40372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":40372},{"type":"v1","addr":"172.21.15.156:6821","nonce":40372}]},"public_addr":"172.21.15.156:6817/40372","cluster_addr":"172.21.15.156:6819/40372","heartbeat_back_addr":"172.21.15.156:6823/40372","heartbeat_front_addr":"172.21.15.156:6821/40372","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:50:12.447063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:50:12.508216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:50:15.509 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-01-16T04:50:15.510 INFO:tasks.ceph:Creating RBD pool 2024-01-16T04:50:15.510 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd pool create rbd 8 2024-01-16T04:50:16.525 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'rbd' created 2024-01-16T04:50:16.538 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-01-16T04:50:17.530 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rbd' on pool 'rbd' 2024-01-16T04:50:17.543 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-16T04:50:17.543 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-16T04:50:17.543 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-16T04:50:17.544 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-16T04:50:17.544 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-16T04:50:17.878 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:50:17.890 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":39935},{"type":"v1","addr":"172.21.15.156:6825","nonce":39935}]},"active_addr":"172.21.15.156:6825/39935","active_change":"2024-01-16T04:50:12.039128+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":3151944062}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":625646200}]}]} 2024-01-16T04:50:17.892 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-16T04:50:17.892 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-16T04:50:17.892 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T04:50:18.177 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:50:18.177 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":12,"fsid":"dae07de8-a7c7-4fa7-b3ca-19c915fbdf1d","created":"2024-01-16T04:50:07.150411+0000","modified":"2024-01-16T04:50:17.524179+0000","last_up_change":"2024-01-16T04:50:14.516683+0000","last_in_change":"2024-01-16T04:50:08.136013+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-16T04:50:13.551691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-16T04:50:15.831542+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2fd312a6-3d18-421f-b641-a81c9bb49196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":40342},{"type":"v1","addr":"172.21.15.156:6801","nonce":40342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":40342},{"type":"v1","addr":"172.21.15.156:6803","nonce":40342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":40342},{"type":"v1","addr":"172.21.15.156:6807","nonce":40342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":40342},{"type":"v1","addr":"172.21.15.156:6805","nonce":40342}]},"public_addr":"172.21.15.156:6801/40342","cluster_addr":"172.21.15.156:6803/40342","heartbeat_back_addr":"172.21.15.156:6807/40342","heartbeat_front_addr":"172.21.15.156:6805/40342","state":["exists","up"]},{"osd":1,"uuid":"9189293d-261d-4048-a9de-6a5778c6ce44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":40343},{"type":"v1","addr":"172.21.15.156:6809","nonce":40343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":40343},{"type":"v1","addr":"172.21.15.156:6811","nonce":40343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":40343},{"type":"v1","addr":"172.21.15.156:6815","nonce":40343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":40343},{"type":"v1","addr":"172.21.15.156:6813","nonce":40343}]},"public_addr":"172.21.15.156:6809/40343","cluster_addr":"172.21.15.156:6811/40343","heartbeat_back_addr":"172.21.15.156:6815/40343","heartbeat_front_addr":"172.21.15.156:6813/40343","state":["exists","up"]},{"osd":2,"uuid":"65c3a83b-fea3-406d-a04b-67e1cdd6856e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":40372},{"type":"v1","addr":"172.21.15.156:6817","nonce":40372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":40372},{"type":"v1","addr":"172.21.15.156:6819","nonce":40372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":40372},{"type":"v1","addr":"172.21.15.156:6823","nonce":40372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":40372},{"type":"v1","addr":"172.21.15.156:6821","nonce":40372}]},"public_addr":"172.21.15.156:6817/40372","cluster_addr":"172.21.15.156:6819/40372","heartbeat_back_addr":"172.21.15.156:6823/40372","heartbeat_front_addr":"172.21.15.156:6821/40372","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:50:12.447063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:50:12.508216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:50:12.525157+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:50:18.190 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-16T04:50:18.190 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T04:50:18.477 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:50:18.477 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":12,"fsid":"dae07de8-a7c7-4fa7-b3ca-19c915fbdf1d","created":"2024-01-16T04:50:07.150411+0000","modified":"2024-01-16T04:50:17.524179+0000","last_up_change":"2024-01-16T04:50:14.516683+0000","last_in_change":"2024-01-16T04:50:08.136013+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-16T04:50:13.551691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-16T04:50:15.831542+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2fd312a6-3d18-421f-b641-a81c9bb49196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":40342},{"type":"v1","addr":"172.21.15.156:6801","nonce":40342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":40342},{"type":"v1","addr":"172.21.15.156:6803","nonce":40342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":40342},{"type":"v1","addr":"172.21.15.156:6807","nonce":40342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":40342},{"type":"v1","addr":"172.21.15.156:6805","nonce":40342}]},"public_addr":"172.21.15.156:6801/40342","cluster_addr":"172.21.15.156:6803/40342","heartbeat_back_addr":"172.21.15.156:6807/40342","heartbeat_front_addr":"172.21.15.156:6805/40342","state":["exists","up"]},{"osd":1,"uuid":"9189293d-261d-4048-a9de-6a5778c6ce44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":40343},{"type":"v1","addr":"172.21.15.156:6809","nonce":40343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":40343},{"type":"v1","addr":"172.21.15.156:6811","nonce":40343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":40343},{"type":"v1","addr":"172.21.15.156:6815","nonce":40343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":40343},{"type":"v1","addr":"172.21.15.156:6813","nonce":40343}]},"public_addr":"172.21.15.156:6809/40343","cluster_addr":"172.21.15.156:6811/40343","heartbeat_back_addr":"172.21.15.156:6815/40343","heartbeat_front_addr":"172.21.15.156:6813/40343","state":["exists","up"]},{"osd":2,"uuid":"65c3a83b-fea3-406d-a04b-67e1cdd6856e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":40372},{"type":"v1","addr":"172.21.15.156:6817","nonce":40372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":40372},{"type":"v1","addr":"172.21.15.156:6819","nonce":40372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":40372},{"type":"v1","addr":"172.21.15.156:6823","nonce":40372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":40372},{"type":"v1","addr":"172.21.15.156:6821","nonce":40372}]},"public_addr":"172.21.15.156:6817/40372","cluster_addr":"172.21.15.156:6819/40372","heartbeat_back_addr":"172.21.15.156:6823/40372","heartbeat_front_addr":"172.21.15.156:6821/40372","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:50:12.447063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:50:12.508216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:50:12.525157+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:50:18.490 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-16T04:50:18.637 INFO:teuthology.orchestra.run.smithi156.stdout:34359738370 2024-01-16T04:50:18.638 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-16T04:50:18.783 INFO:teuthology.orchestra.run.smithi156.stdout:34359738370 2024-01-16T04:50:18.783 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-16T04:50:18.929 INFO:teuthology.orchestra.run.smithi156.stdout:38654705668 2024-01-16T04:50:18.929 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T04:50:19.217 INFO:teuthology.orchestra.run.smithi156.stdout:34359738369 2024-01-16T04:50:19.230 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.0 2024-01-16T04:50:20.231 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T04:50:20.530 INFO:teuthology.orchestra.run.smithi156.stdout:34359738371 2024-01-16T04:50:20.544 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738371 for osd.0 2024-01-16T04:50:20.544 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-16T04:50:20.831 INFO:teuthology.orchestra.run.smithi156.stdout:34359738371 2024-01-16T04:50:20.844 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738371 for osd.1 2024-01-16T04:50:20.845 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-16T04:50:21.129 INFO:teuthology.orchestra.run.smithi156.stdout:38654705668 2024-01-16T04:50:21.142 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.2 2024-01-16T04:50:21.142 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-16T04:50:21.142 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T04:50:21.423 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:50:21.425 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-01-16T04:50:21.436 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-01-16T04:50:20.043875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":891608,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":313681192,"statfs":{"total":322122547200,"available":321209540608,"internally_reserved":0,"allocated":327680,"data_stored":91285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.532314+0000","last_change":"2024-01-16T04:50:17.531756+0000","last_active":"2024-01-16T04:50:17.532314+0000","last_peered":"2024-01-16T04:50:17.532314+0000","last_clean":"2024-01-16T04:50:17.532314+0000","last_became_active":"2024-01-16T04:50:17.531483+0000","last_became_peered":"2024-01-16T04:50:17.531483+0000","last_unstale":"2024-01-16T04:50:17.532314+0000","last_undegraded":"2024-01-16T04:50:17.532314+0000","last_fullsized":"2024-01-16T04:50:17.532314+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_clean_scrub_stamp":"2024-01-16T04:50:16.521940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.533353+0000","last_change":"2024-01-16T04:50:17.532713+0000","last_active":"2024-01-16T04:50:17.533353+0000","last_peered":"2024-01-16T04:50:17.533353+0000","last_clean":"2024-01-16T04:50:17.533353+0000","last_became_active":"2024-01-16T04:50:17.532378+0000","last_became_peered":"2024-01-16T04:50:17.532378+0000","last_unstale":"2024-01-16T04:50:17.533353+0000","last_undegraded":"2024-01-16T04:50:17.533353+0000","last_fullsized":"2024-01-16T04:50:17.533353+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_clean_scrub_stamp":"2024-01-16T04:50:16.521940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.533360+0000","last_change":"2024-01-16T04:50:17.532619+0000","last_active":"2024-01-16T04:50:17.533360+0000","last_peered":"2024-01-16T04:50:17.533360+0000","last_clean":"2024-01-16T04:50:17.533360+0000","last_became_active":"2024-01-16T04:50:17.532354+0000","last_became_peered":"2024-01-16T04:50:17.532354+0000","last_unstale":"2024-01-16T04:50:17.533360+0000","last_undegraded":"2024-01-16T04:50:17.533360+0000","last_fullsized":"2024-01-16T04:50:17.533360+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_clean_scrub_stamp":"2024-01-16T04:50:16.521940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.530567+0000","last_change":"2024-01-16T04:50:17.530567+0000","last_active":"2024-01-16T04:50:17.530567+0000","last_peered":"2024-01-16T04:50:17.530567+0000","last_clean":"2024-01-16T04:50:17.530567+0000","last_became_active":"2024-01-16T04:50:17.530258+0000","last_became_peered":"2024-01-16T04:50:17.530258+0000","last_unstale":"2024-01-16T04:50:17.530567+0000","last_undegraded":"2024-01-16T04:50:17.530567+0000","last_fullsized":"2024-01-16T04:50:17.530567+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_clean_scrub_stamp":"2024-01-16T04:50:16.521940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.528631+0000","last_change":"2024-01-16T04:50:15.523621+0000","last_active":"2024-01-16T04:50:17.528631+0000","last_peered":"2024-01-16T04:50:17.528631+0000","last_clean":"2024-01-16T04:50:17.528631+0000","last_became_active":"2024-01-16T04:50:15.523405+0000","last_became_peered":"2024-01-16T04:50:15.523405+0000","last_unstale":"2024-01-16T04:50:17.528631+0000","last_undegraded":"2024-01-16T04:50:17.528631+0000","last_fullsized":"2024-01-16T04:50:17.528631+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:14.516683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:14.516683+0000","last_clean_scrub_stamp":"2024-01-16T04:50:14.516683+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.532319+0000","last_change":"2024-01-16T04:50:17.531317+0000","last_active":"2024-01-16T04:50:17.532319+0000","last_peered":"2024-01-16T04:50:17.532319+0000","last_clean":"2024-01-16T04:50:17.532319+0000","last_became_active":"2024-01-16T04:50:17.531038+0000","last_became_peered":"2024-01-16T04:50:17.531038+0000","last_unstale":"2024-01-16T04:50:17.532319+0000","last_undegraded":"2024-01-16T04:50:17.532319+0000","last_fullsized":"2024-01-16T04:50:17.532319+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_clean_scrub_stamp":"2024-01-16T04:50:16.521940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.532035+0000","last_change":"2024-01-16T04:50:17.532035+0000","last_active":"2024-01-16T04:50:17.532035+0000","last_peered":"2024-01-16T04:50:17.532035+0000","last_clean":"2024-01-16T04:50:17.532035+0000","last_became_active":"2024-01-16T04:50:17.531850+0000","last_became_peered":"2024-01-16T04:50:17.531850+0000","last_unstale":"2024-01-16T04:50:17.532035+0000","last_undegraded":"2024-01-16T04:50:17.532035+0000","last_fullsized":"2024-01-16T04:50:17.532035+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_clean_scrub_stamp":"2024-01-16T04:50:16.521940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.532919+0000","last_change":"2024-01-16T04:50:17.532919+0000","last_active":"2024-01-16T04:50:17.532919+0000","last_peered":"2024-01-16T04:50:17.532919+0000","last_clean":"2024-01-16T04:50:17.532919+0000","last_became_active":"2024-01-16T04:50:17.532661+0000","last_became_peered":"2024-01-16T04:50:17.532661+0000","last_unstale":"2024-01-16T04:50:17.532919+0000","last_undegraded":"2024-01-16T04:50:17.532919+0000","last_fullsized":"2024-01-16T04:50:17.532919+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_clean_scrub_stamp":"2024-01-16T04:50:16.521940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.532332+0000","last_change":"2024-01-16T04:50:17.531751+0000","last_active":"2024-01-16T04:50:17.532332+0000","last_peered":"2024-01-16T04:50:17.532332+0000","last_clean":"2024-01-16T04:50:17.532332+0000","last_became_active":"2024-01-16T04:50:17.531438+0000","last_became_peered":"2024-01-16T04:50:17.531438+0000","last_unstale":"2024-01-16T04:50:17.532332+0000","last_undegraded":"2024-01-16T04:50:17.532332+0000","last_fullsized":"2024-01-16T04:50:17.532332+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_clean_scrub_stamp":"2024-01-16T04:50:16.521940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":29252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":29252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560392,"statfs":{"total":107374182400,"available":107069841408,"internally_reserved":0,"allocated":114688,"data_stored":32781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T04:50:21.437 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T04:50:21.717 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:50:21.719 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-01-16T04:50:21.730 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-01-16T04:50:20.043875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":891608,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":313681192,"statfs":{"total":322122547200,"available":321209540608,"internally_reserved":0,"allocated":327680,"data_stored":91285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.532314+0000","last_change":"2024-01-16T04:50:17.531756+0000","last_active":"2024-01-16T04:50:17.532314+0000","last_peered":"2024-01-16T04:50:17.532314+0000","last_clean":"2024-01-16T04:50:17.532314+0000","last_became_active":"2024-01-16T04:50:17.531483+0000","last_became_peered":"2024-01-16T04:50:17.531483+0000","last_unstale":"2024-01-16T04:50:17.532314+0000","last_undegraded":"2024-01-16T04:50:17.532314+0000","last_fullsized":"2024-01-16T04:50:17.532314+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_clean_scrub_stamp":"2024-01-16T04:50:16.521940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.533353+0000","last_change":"2024-01-16T04:50:17.532713+0000","last_active":"2024-01-16T04:50:17.533353+0000","last_peered":"2024-01-16T04:50:17.533353+0000","last_clean":"2024-01-16T04:50:17.533353+0000","last_became_active":"2024-01-16T04:50:17.532378+0000","last_became_peered":"2024-01-16T04:50:17.532378+0000","last_unstale":"2024-01-16T04:50:17.533353+0000","last_undegraded":"2024-01-16T04:50:17.533353+0000","last_fullsized":"2024-01-16T04:50:17.533353+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_clean_scrub_stamp":"2024-01-16T04:50:16.521940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.533360+0000","last_change":"2024-01-16T04:50:17.532619+0000","last_active":"2024-01-16T04:50:17.533360+0000","last_peered":"2024-01-16T04:50:17.533360+0000","last_clean":"2024-01-16T04:50:17.533360+0000","last_became_active":"2024-01-16T04:50:17.532354+0000","last_became_peered":"2024-01-16T04:50:17.532354+0000","last_unstale":"2024-01-16T04:50:17.533360+0000","last_undegraded":"2024-01-16T04:50:17.533360+0000","last_fullsized":"2024-01-16T04:50:17.533360+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_clean_scrub_stamp":"2024-01-16T04:50:16.521940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.530567+0000","last_change":"2024-01-16T04:50:17.530567+0000","last_active":"2024-01-16T04:50:17.530567+0000","last_peered":"2024-01-16T04:50:17.530567+0000","last_clean":"2024-01-16T04:50:17.530567+0000","last_became_active":"2024-01-16T04:50:17.530258+0000","last_became_peered":"2024-01-16T04:50:17.530258+0000","last_unstale":"2024-01-16T04:50:17.530567+0000","last_undegraded":"2024-01-16T04:50:17.530567+0000","last_fullsized":"2024-01-16T04:50:17.530567+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_clean_scrub_stamp":"2024-01-16T04:50:16.521940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.528631+0000","last_change":"2024-01-16T04:50:15.523621+0000","last_active":"2024-01-16T04:50:17.528631+0000","last_peered":"2024-01-16T04:50:17.528631+0000","last_clean":"2024-01-16T04:50:17.528631+0000","last_became_active":"2024-01-16T04:50:15.523405+0000","last_became_peered":"2024-01-16T04:50:15.523405+0000","last_unstale":"2024-01-16T04:50:17.528631+0000","last_undegraded":"2024-01-16T04:50:17.528631+0000","last_fullsized":"2024-01-16T04:50:17.528631+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:14.516683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:14.516683+0000","last_clean_scrub_stamp":"2024-01-16T04:50:14.516683+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.532319+0000","last_change":"2024-01-16T04:50:17.531317+0000","last_active":"2024-01-16T04:50:17.532319+0000","last_peered":"2024-01-16T04:50:17.532319+0000","last_clean":"2024-01-16T04:50:17.532319+0000","last_became_active":"2024-01-16T04:50:17.531038+0000","last_became_peered":"2024-01-16T04:50:17.531038+0000","last_unstale":"2024-01-16T04:50:17.532319+0000","last_undegraded":"2024-01-16T04:50:17.532319+0000","last_fullsized":"2024-01-16T04:50:17.532319+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_clean_scrub_stamp":"2024-01-16T04:50:16.521940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.532035+0000","last_change":"2024-01-16T04:50:17.532035+0000","last_active":"2024-01-16T04:50:17.532035+0000","last_peered":"2024-01-16T04:50:17.532035+0000","last_clean":"2024-01-16T04:50:17.532035+0000","last_became_active":"2024-01-16T04:50:17.531850+0000","last_became_peered":"2024-01-16T04:50:17.531850+0000","last_unstale":"2024-01-16T04:50:17.532035+0000","last_undegraded":"2024-01-16T04:50:17.532035+0000","last_fullsized":"2024-01-16T04:50:17.532035+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_clean_scrub_stamp":"2024-01-16T04:50:16.521940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.532919+0000","last_change":"2024-01-16T04:50:17.532919+0000","last_active":"2024-01-16T04:50:17.532919+0000","last_peered":"2024-01-16T04:50:17.532919+0000","last_clean":"2024-01-16T04:50:17.532919+0000","last_became_active":"2024-01-16T04:50:17.532661+0000","last_became_peered":"2024-01-16T04:50:17.532661+0000","last_unstale":"2024-01-16T04:50:17.532919+0000","last_undegraded":"2024-01-16T04:50:17.532919+0000","last_fullsized":"2024-01-16T04:50:17.532919+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_clean_scrub_stamp":"2024-01-16T04:50:16.521940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-16T04:50:17.532332+0000","last_change":"2024-01-16T04:50:17.531751+0000","last_active":"2024-01-16T04:50:17.532332+0000","last_peered":"2024-01-16T04:50:17.532332+0000","last_clean":"2024-01-16T04:50:17.532332+0000","last_became_active":"2024-01-16T04:50:17.531438+0000","last_became_peered":"2024-01-16T04:50:17.531438+0000","last_unstale":"2024-01-16T04:50:17.532332+0000","last_undegraded":"2024-01-16T04:50:17.532332+0000","last_fullsized":"2024-01-16T04:50:17.532332+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T04:50:16.521940+0000","last_clean_scrub_stamp":"2024-01-16T04:50:16.521940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":29252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":29252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560392,"statfs":{"total":107374182400,"available":107069841408,"internally_reserved":0,"allocated":114688,"data_stored":32781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T04:50:21.731 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-16T04:50:21.731 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-16T04:50:21.731 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-16T04:50:21.731 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-16T04:50:22.050 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:50:22.064 INFO:teuthology.orchestra.run.smithi156.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-16T04:50:22.064 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-16T04:50:22.064 INFO:teuthology.run_tasks:Running task vault... 2024-01-16T04:50:22.080 DEBUG:tasks.vault:Vault config is {'client.0': {'engine': 'transit', 'install_sha256': '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458', 'install_url': 'https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip', 'prefix': '/v1/transit/keys/', 'root_token': 'test_root_token', 'secrets': [{'path': 'my-key-1'}, {'path': 'my-key-2'}]}} 2024-01-16T04:50:22.080 INFO:tasks.vault:Downloading Vault... 2024-01-16T04:50:22.080 INFO:tasks.vault:Downloading Vault... 2024-01-16T04:50:22.081 DEBUG:teuthology.orchestra.run.smithi156:> curl -L https://releases.hashicorp.com/vault/1.2.2/vault_1.2.2_linux_amd64.zip -o /home/ubuntu/cephtest/vault.zip 2024-01-16T04:50:22.110 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:50:22.111 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:50:23.356 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 9 42.6M 9 4014k 0 0 8327k 0 0:00:05 --:--:-- 0:00:05 8310k 100 42.6M 100 42.6M 0 0 34.2M 0 0:00:01 0:00:01 --:--:-- 34.2M 2024-01-16T04:50:23.357 INFO:tasks.vault:Verifying SHA256 signature... 2024-01-16T04:50:23.357 DEBUG:teuthology.orchestra.run.smithi156:> echo '7725b35d9ca8be3668abe63481f0731ca4730509419b4eb29fa0b0baa4798458 /home/ubuntu/cephtest/vault.zip' | sha256sum --check --status 2024-01-16T04:50:23.523 INFO:tasks.vault:Extracting vault... 2024-01-16T04:50:23.523 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p /home/ubuntu/cephtest/vault 2024-01-16T04:50:23.540 DEBUG:teuthology.orchestra.run.smithi156:> unzip /home/ubuntu/cephtest/vault.zip -d /home/ubuntu/cephtest/vault 2024-01-16T04:50:23.605 INFO:teuthology.orchestra.run.smithi156.stdout:Archive: /home/ubuntu/cephtest/vault.zip 2024-01-16T04:50:24.327 INFO:teuthology.orchestra.run.smithi156.stdout: inflating: /home/ubuntu/cephtest/vault/vault 2024-01-16T04:50:24.327 INFO:tasks.vault:Starting Vault listening on 0.0.0.0:8200 ... 2024-01-16T04:50:24.327 INFO:tasks.vault.client.0:Restarting daemon 2024-01-16T04:50:24.328 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest/vault && exec bash -c 'chmod +x /home/ubuntu/cephtest/vault/vault && /home/ubuntu/cephtest/vault/vault server -dev -dev-listen-address=0.0.0.0:8200 -dev-no-store-token -dev-root-token-id=test_root_token' & { read; kill %1; }) 2024-01-16T04:50:24.370 INFO:tasks.vault.client.0:Started 2024-01-16T04:50:24.419 INFO:tasks.vault.client.0.smithi156.stdout:==> Vault server configuration: 2024-01-16T04:50:24.419 INFO:tasks.vault.client.0.smithi156.stdout: 2024-01-16T04:50:24.419 INFO:tasks.vault.client.0.smithi156.stdout: Api Address: http://0.0.0.0:8200 2024-01-16T04:50:24.419 INFO:tasks.vault.client.0.smithi156.stdout: Cgo: disabled 2024-01-16T04:50:24.419 INFO:tasks.vault.client.0.smithi156.stdout: Cluster Address: https://0.0.0.0:8201 2024-01-16T04:50:24.420 INFO:tasks.vault.client.0.smithi156.stdout: Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled") 2024-01-16T04:50:24.420 INFO:tasks.vault.client.0.smithi156.stdout: Log Level: info 2024-01-16T04:50:24.420 INFO:tasks.vault.client.0.smithi156.stdout: Mlock: supported: true, enabled: false 2024-01-16T04:50:24.420 INFO:tasks.vault.client.0.smithi156.stdout: Storage: inmem 2024-01-16T04:50:24.420 INFO:tasks.vault.client.0.smithi156.stdout: Version: Vault v1.2.2 2024-01-16T04:50:24.420 INFO:tasks.vault.client.0.smithi156.stdout: 2024-01-16T04:50:24.436 INFO:tasks.vault.client.0.smithi156.stdout:WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory 2024-01-16T04:50:24.436 INFO:tasks.vault.client.0.smithi156.stdout:and starts unsealed with a single unseal key. The root token is already 2024-01-16T04:50:24.437 INFO:tasks.vault.client.0.smithi156.stdout:authenticated to the CLI, so you can immediately begin using Vault. 2024-01-16T04:50:24.437 INFO:tasks.vault.client.0.smithi156.stdout: 2024-01-16T04:50:24.437 INFO:tasks.vault.client.0.smithi156.stdout:You may need to set the following environment variable: 2024-01-16T04:50:24.437 INFO:tasks.vault.client.0.smithi156.stdout: 2024-01-16T04:50:24.437 INFO:tasks.vault.client.0.smithi156.stdout: $ export VAULT_ADDR='http://0.0.0.0:8200' 2024-01-16T04:50:24.437 INFO:tasks.vault.client.0.smithi156.stdout: 2024-01-16T04:50:24.437 INFO:tasks.vault.client.0.smithi156.stdout:The unseal key and root token are displayed below in case you want to 2024-01-16T04:50:24.437 INFO:tasks.vault.client.0.smithi156.stdout:seal/unseal the Vault or re-authenticate. 2024-01-16T04:50:24.437 INFO:tasks.vault.client.0.smithi156.stdout: 2024-01-16T04:50:24.437 INFO:tasks.vault.client.0.smithi156.stdout:Unseal Key: 7S6XcY4FGwnUPtIqtAqs+m+3Hr5ZJoF4QecAyfP8fKs= 2024-01-16T04:50:24.437 INFO:tasks.vault.client.0.smithi156.stdout:Root Token: test_root_token 2024-01-16T04:50:24.438 INFO:tasks.vault.client.0.smithi156.stdout: 2024-01-16T04:50:24.438 INFO:tasks.vault.client.0.smithi156.stdout:Development mode should NOT be used in production installations! 2024-01-16T04:50:24.438 INFO:tasks.vault.client.0.smithi156.stdout: 2024-01-16T04:50:24.438 INFO:tasks.vault.client.0.smithi156.stdout:==> Vault server started! Log data will stream in below: 2024-01-16T04:50:24.438 INFO:tasks.vault.client.0.smithi156.stdout: 2024-01-16T04:50:24.438 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.418Z [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set 2024-01-16T04:50:24.438 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.419Z [INFO] core: security barrier not initialized 2024-01-16T04:50:24.438 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.419Z [INFO] core: security barrier initialized: shares=1 threshold=1 2024-01-16T04:50:24.439 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.419Z [INFO] core: post-unseal setup starting 2024-01-16T04:50:24.439 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.427Z [INFO] core: loaded wrapping token key 2024-01-16T04:50:24.439 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.427Z [INFO] core: successfully setup plugin catalog: plugin-directory= 2024-01-16T04:50:24.439 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.427Z [INFO] core: no mounts; adding default mount table 2024-01-16T04:50:24.439 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.428Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2024-01-16T04:50:24.439 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.428Z [INFO] core: successfully mounted backend: type=system path=sys/ 2024-01-16T04:50:24.439 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.429Z [INFO] core: successfully mounted backend: type=identity path=identity/ 2024-01-16T04:50:24.439 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.430Z [INFO] core: successfully enabled credential backend: type=token path=token/ 2024-01-16T04:50:24.439 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.430Z [INFO] core: restoring leases 2024-01-16T04:50:24.439 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.430Z [INFO] rollback: starting rollback manager 2024-01-16T04:50:24.439 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.430Z [INFO] expiration: lease restore complete 2024-01-16T04:50:24.439 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.430Z [INFO] identity: entities restored 2024-01-16T04:50:24.439 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.430Z [INFO] identity: groups restored 2024-01-16T04:50:24.439 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.430Z [INFO] core: post-unseal setup complete 2024-01-16T04:50:24.439 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.431Z [INFO] core: root token generated 2024-01-16T04:50:24.440 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.431Z [INFO] core: pre-seal teardown starting 2024-01-16T04:50:24.440 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.431Z [INFO] rollback: stopping rollback manager 2024-01-16T04:50:24.440 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.431Z [INFO] core: pre-seal teardown complete 2024-01-16T04:50:24.440 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.431Z [INFO] core.cluster-listener: starting listener: listener_address=0.0.0.0:8201 2024-01-16T04:50:24.440 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.431Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201 2024-01-16T04:50:24.440 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.431Z [INFO] core: post-unseal setup starting 2024-01-16T04:50:24.440 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.431Z [INFO] core: loaded wrapping token key 2024-01-16T04:50:24.440 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.431Z [INFO] core: successfully setup plugin catalog: plugin-directory= 2024-01-16T04:50:24.440 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.431Z [INFO] core: successfully mounted backend: type=system path=sys/ 2024-01-16T04:50:24.440 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.432Z [INFO] core: successfully mounted backend: type=identity path=identity/ 2024-01-16T04:50:24.440 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.432Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/ 2024-01-16T04:50:24.440 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.432Z [INFO] core: successfully enabled credential backend: type=token path=token/ 2024-01-16T04:50:24.440 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.432Z [INFO] core: restoring leases 2024-01-16T04:50:24.440 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.432Z [INFO] rollback: starting rollback manager 2024-01-16T04:50:24.441 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.432Z [INFO] identity: entities restored 2024-01-16T04:50:24.441 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.432Z [INFO] identity: groups restored 2024-01-16T04:50:24.441 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.432Z [INFO] core: post-unseal setup complete 2024-01-16T04:50:24.441 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.432Z [INFO] core: vault is unsealed 2024-01-16T04:50:24.441 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.432Z [INFO] expiration: lease restore complete 2024-01-16T04:50:24.441 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.433Z [INFO] expiration: revoked lease: lease_id=auth/token/root/h0b22f24a7bfb2419c78270e5ec32a9ed8c21c2660db417cef38fd141edb90152 2024-01-16T04:50:24.441 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.434Z [INFO] core: successful mount: namespace= path=secret/ type=kv 2024-01-16T04:50:24.441 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.444Z [INFO] secrets.kv.kv_2e3fbf7f: collecting keys to upgrade 2024-01-16T04:50:24.442 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.444Z [INFO] secrets.kv.kv_2e3fbf7f: done collecting keys: num_keys=1 2024-01-16T04:50:24.442 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:24.444Z [INFO] secrets.kv.kv_2e3fbf7f: upgrading keys finished 2024-01-16T04:50:34.370 INFO:tasks.vault:Mounting transit secrets engine 2024-01-16T04:50:34.371 INFO:tasks.vault:Send request to Vault: smithi156.front.sepia.ceph.com:8200 at /v1/sys/mounts/transit with token: test_root_token 2024-01-16T04:50:34.377 INFO:tasks.vault:b'' 2024-01-16T04:50:34.377 INFO:tasks.vault:Send request to Vault: smithi156.front.sepia.ceph.com:8200 at /v1/transit/keys/my-key-1 with token: test_root_token 2024-01-16T04:50:34.378 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:50:34.381Z [INFO] core: successful mount: namespace= path=transit/ type=transit 2024-01-16T04:50:34.380 INFO:tasks.vault:b'' 2024-01-16T04:50:34.380 INFO:tasks.vault:Send request to Vault: smithi156.front.sepia.ceph.com:8200 at /v1/transit/keys/my-key-2 with token: test_root_token 2024-01-16T04:50:34.384 INFO:tasks.vault:b'' 2024-01-16T04:50:34.384 INFO:tasks.vault:secrets created 2024-01-16T04:50:34.384 INFO:teuthology.run_tasks:Running task rgw... 2024-01-16T04:50:34.398 DEBUG:tasks.rgw:config is {'client.0': {'use-vault-role': 'client.0'}} 2024-01-16T04:50:34.398 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2024-01-16T04:50:34.398 INFO:tasks.rgw:Creating data pools 2024-01-16T04:50:34.398 DEBUG:tasks.rgw:Obtaining remote for client client.0 2024-01-16T04:50:34.398 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2024-01-16T04:50:35.135 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'default.rgw.buckets.data' created 2024-01-16T04:50:35.150 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2024-01-16T04:50:36.153 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2024-01-16T04:50:36.168 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2024-01-16T04:50:37.137 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'default.rgw.buckets.index' created 2024-01-16T04:50:37.153 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2024-01-16T04:50:38.151 INFO:teuthology.orchestra.run.smithi156.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2024-01-16T04:50:38.170 DEBUG:tasks.rgw:Pools created 2024-01-16T04:50:38.170 INFO:tasks.rgw:Starting rgw... 2024-01-16T04:50:38.170 INFO:tasks.rgw:rgw client.0 config is {'use-vault-role': 'client.0'} 2024-01-16T04:50:38.171 INFO:tasks.rgw:Using civetweb as radosgw frontend 2024-01-16T04:50:38.171 DEBUG:teuthology.orchestra.run.smithi156:> sudo echo -n test_root_token | sudo tee /etc/ceph/vault-root-token 2024-01-16T04:50:38.200 INFO:teuthology.orchestra.run.smithi156.stdout:test_root_token 2024-01-16T04:50:38.201 INFO:tasks.rgw:Token file content 2024-01-16T04:50:38.201 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/ceph/vault-root-token 2024-01-16T04:50:38.260 INFO:teuthology.orchestra.run.smithi156.stdout:test_root_token 2024-01-16T04:50:38.260 INFO:tasks.rgw:Restrict access to token file 2024-01-16T04:50:38.260 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 600 /etc/ceph/vault-root-token 2024-01-16T04:50:38.328 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown ceph /etc/ceph/vault-root-token 2024-01-16T04:50:38.396 INFO:tasks.rgw.client.0:Restarting daemon 2024-01-16T04:50:38.396 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_vault_addr smithi156.front.sepia.ceph.com:8200 --rgw_crypt_vault_token_file /etc/ceph/vault-root-token --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2024-01-16T04:50:38.439 INFO:tasks.rgw.client.0:Started 2024-01-16T04:50:38.439 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi156.front.sepia.ceph.com:80/ 2024-01-16T04:50:38.439 DEBUG:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:80/ 2024-01-16T04:50:38.464 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-01-16T04:50:38.465 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:50:38.465 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:50:38.465 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi156.front.sepia.ceph.com port 80: Connection refused 2024-01-16T04:50:39.466 DEBUG:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:80/ 2024-01-16T04:50:39.491 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:50:39.491 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:50:39.493 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-01-16T04:50:39.493 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi156.front.sepia.ceph.com port 80: Connection refused 2024-01-16T04:50:41.495 DEBUG:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:80/ 2024-01-16T04:50:41.521 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:50:41.521 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:50:41.523 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-01-16T04:50:41.523 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi156.front.sepia.ceph.com port 80: Connection refused 2024-01-16T04:50:45.524 DEBUG:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:80/ 2024-01-16T04:50:45.551 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:50:45.551 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:50:45.551 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi156.front.sepia.ceph.com port 80: Connection refused 2024-01-16T04:50:45.552 DEBUG:teuthology.orchestra.run:got remote process result: 7 2024-01-16T04:50:47.219 INFO:tasks.rgw.client.0.smithi156.stdout:2024-01-16T04:50:47.222+0000 7f30e6391880 -1 IMPORTANT: the civetweb frontend is now deprecated and will be removed in a future release 2024-01-16T04:50:53.553 DEBUG:teuthology.orchestra.run.smithi156:> curl http://smithi156.front.sepia.ceph.com:80/ 2024-01-16T04:50:53.580 INFO:teuthology.orchestra.run.smithi156.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2024-01-16T04:50:53.580 INFO:teuthology.orchestra.run.smithi156.stderr: Dload Upload Total Spent Left Speed 2024-01-16T04:50:53.580 INFO:teuthology.orchestra.run.smithi156.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2024-01-16T04:50:53.581 INFO:teuthology.orchestra.run.smithi156.stdout:anonymous 2024-01-16T04:50:53.582 INFO:teuthology.run_tasks:Running task s3tests... 2024-01-16T04:50:53.596 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-pacific', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}} 2024-01-16T04:50:53.597 INFO:tasks.s3tests:Downloading s3-tests... 2024-01-16T04:50:53.597 INFO:tasks.s3tests:Using branch 'ceph-pacific' for s3tests 2024-01-16T04:50:53.597 DEBUG:teuthology.orchestra.run.smithi156:> git clone -b ceph-pacific https://git.ceph.com/s3-tests.git /home/ubuntu/cephtest/s3-tests 2024-01-16T04:50:53.642 INFO:teuthology.orchestra.run.smithi156.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests'... 2024-01-16T04:50:54.353 INFO:tasks.s3tests:Creating rgw users... 2024-01-16T04:50:54.353 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2024-01-16T04:50:54.353 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key JCZHIUSFADNDMNFEHUYO --secret JNplZ8RWGrssQX6YvS4uNwsAXKSsThpUN1Mt3ndCTy/ejw9Mhl+W1A== --email foo.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2024-01-16T04:50:54.442 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:50:54.443 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:50:54.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.451+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.452+0000 7f60c2bcc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.452+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.452+0000 7f60c2bcc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.452+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.452+0000 7f60c2bcc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.452+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.453+0000 7f60c2bcc7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:54.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.459+0000 7f60c2bcc7c0 20 RGWRados::pool_iterate: got zonegroup_info.92d3822f-5de2-431f-9d8f-b46ea51d44b6 2024-01-16T04:50:54.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.459+0000 7f60c2bcc7c0 20 RGWRados::pool_iterate: got zone_info.2259a717-7b4f-4cc8-b097-8339a13202a0 2024-01-16T04:50:54.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.459+0000 7f60c2bcc7c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:50:54.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.459+0000 7f60c2bcc7c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:50:54.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.459+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.459+0000 7f60c2bcc7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:54.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.459+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.460+0000 7f60c2bcc7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:50:54.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.460+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.460+0000 7f60c2bcc7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:54.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.460+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.460+0000 7f60c2bcc7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:50:54.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.460+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.460+0000 7f60c2bcc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.460+0000 7f60c2bcc7c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:50:54.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.460+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.461+0000 7f60c2bcc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.461+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.461+0000 7f60c2bcc7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:54.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.461+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.461+0000 7f60c2bcc7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:50:54.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.461+0000 7f60c2bcc7c0 20 zonegroup default 2024-01-16T04:50:54.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.461+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.461+0000 7f60c2bcc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.461+0000 7f60c2bcc7c0 10 Cannot find current period zone using local zone 2024-01-16T04:50:54.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.461+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.461+0000 7f60c2bcc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.461+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.462+0000 7f60c2bcc7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:54.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.462+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.462+0000 7f60c2bcc7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:50:54.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.462+0000 7f60c2bcc7c0 20 zone default found 2024-01-16T04:50:54.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.462+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.463 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.462+0000 7f60c2bcc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.463 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.462+0000 7f60c2bcc7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.463 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.462+0000 7f60c2bcc7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.463 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.462+0000 7f60c2bcc7c0 20 started sync module instance, tier type = 2024-01-16T04:50:54.463 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.462+0000 7f60c2bcc7c0 20 started zone id=2259a717-7b4f-4cc8-b097-8339a13202a0 (name=default) with tier type = 2024-01-16T04:50:54.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.469+0000 7f60c2bcc7c0 20 add_watcher() i=0 2024-01-16T04:50:54.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.469+0000 7f60c2bcc7c0 20 add_watcher() i=1 2024-01-16T04:50:54.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.469+0000 7f60c2bcc7c0 20 add_watcher() i=2 2024-01-16T04:50:54.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.469+0000 7f60c2bcc7c0 20 add_watcher() i=3 2024-01-16T04:50:54.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.469+0000 7f60c2bcc7c0 20 add_watcher() i=4 2024-01-16T04:50:54.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.469+0000 7f60c2bcc7c0 20 add_watcher() i=5 2024-01-16T04:50:54.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.469+0000 7f60c2bcc7c0 20 add_watcher() i=6 2024-01-16T04:50:54.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.470+0000 7f60c2bcc7c0 20 add_watcher() i=7 2024-01-16T04:50:54.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.470+0000 7f60c2bcc7c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:50:54.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.472+0000 7f607f7de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:50:54.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.472+0000 7f60c2bcc7c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:50:54.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.472+0000 7f60c2bcc7c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:50:54.473 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.476+0000 7f60c2bcc7c0 5 note: GC not initialized 2024-01-16T04:50:54.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.477+0000 7f5fee7fc700 20 reqs_thread_entry: start 2024-01-16T04:50:54.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.477+0000 7f5fdffff700 20 reqs_thread_entry: start 2024-01-16T04:50:54.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.477+0000 7f60c2bcc7c0 20 init_complete bucket index max shards: 11 2024-01-16T04:50:54.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.477+0000 7f60c2bcc7c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:50:54.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.477+0000 7f5fde7fc700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:51:24 2024 2024-01-16T04:50:54.475 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:50:54.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.477+0000 7f5fddffb700 20 reqs_thread_entry: start 2024-01-16T04:50:54.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.477+0000 7f60c2bcc7c0 20 get_system_obj_state: rctx=0x7fff53dfddb8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x556512113d50 s->prefetch_data=0 2024-01-16T04:50:54.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.477+0000 7f60c2bcc7c0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2024-01-16T04:50:54.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.477+0000 7f60c2bcc7c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2024-01-16T04:50:54.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.477+0000 7f60c2bcc7c0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:50:54.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.477+0000 7f60c2bcc7c0 20 get_system_obj_state: rctx=0x7fff53dfddc8 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x556512113d50 s->prefetch_data=0 2024-01-16T04:50:54.476 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.477+0000 7f60c2bcc7c0 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2024-01-16T04:50:54.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.478+0000 7f60c2bcc7c0 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2024-01-16T04:50:54.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.478+0000 7f60c2bcc7c0 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2024-01-16T04:50:54.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.478+0000 7f60c2bcc7c0 20 get_system_obj_state: rctx=0x7fff53dfddc8 obj=default.rgw.meta:users.keys:JCZHIUSFADNDMNFEHUYO state=0x556512113d50 s->prefetch_data=0 2024-01-16T04:50:54.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.478+0000 7f60c2bcc7c0 10 cache get: name=default.rgw.meta+users.keys+JCZHIUSFADNDMNFEHUYO : miss 2024-01-16T04:50:54.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.478+0000 7f60c2bcc7c0 10 cache put: name=default.rgw.meta+users.keys+JCZHIUSFADNDMNFEHUYO info.flags=0x0 2024-01-16T04:50:54.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.478+0000 7f60c2bcc7c0 10 adding default.rgw.meta+users.keys+JCZHIUSFADNDMNFEHUYO to cache LRU end 2024-01-16T04:50:54.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.478+0000 7f60c2bcc7c0 20 get_system_obj_state: rctx=0x7fff53dfd6e8 obj=default.rgw.meta:users.keys:JCZHIUSFADNDMNFEHUYO state=0x556512113d50 s->prefetch_data=0 2024-01-16T04:50:54.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.478+0000 7f60c2bcc7c0 10 cache get: name=default.rgw.meta+users.keys+JCZHIUSFADNDMNFEHUYO : hit (negative entry) 2024-01-16T04:50:54.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.478+0000 7f60c2bcc7c0 20 get_system_obj_state: rctx=0x7fff53dfda28 obj=default.rgw.meta:users.keys:JCZHIUSFADNDMNFEHUYO state=0x556512113d50 s->prefetch_data=0 2024-01-16T04:50:54.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.478+0000 7f60c2bcc7c0 10 cache get: name=default.rgw.meta+users.keys+JCZHIUSFADNDMNFEHUYO : hit (negative entry) 2024-01-16T04:50:54.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.479+0000 7f60c2bcc7c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-01-16T04:50:54.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.479+0000 7f60c2bcc7c0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:50:54.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.479+0000 7f60c2bcc7c0 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2024-01-16T04:50:54.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.479+0000 7f609f7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149696 cookie 93892584496480 notifier 4199 bl.length()=640 2024-01-16T04:50:54.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.479+0000 7f609f7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-01-16T04:50:54.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.479+0000 7f609f7fe700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:50:54.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.481+0000 7f60c2bcc7c0 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2024-01-16T04:50:54.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.481+0000 7f60c2bcc7c0 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2024-01-16T04:50:54.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.481+0000 7f60c2bcc7c0 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2024-01-16T04:50:54.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.481+0000 7f609effd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149696 cookie 93892584513472 notifier 4199 bl.length()=194 2024-01-16T04:50:54.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.481+0000 7f609effd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2024-01-16T04:50:54.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.481+0000 7f609effd700 10 rgw watcher librados: moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2024-01-16T04:50:54.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.482+0000 7f60c2bcc7c0 10 cache put: name=default.rgw.meta+users.keys+JCZHIUSFADNDMNFEHUYO info.flags=0x7 2024-01-16T04:50:54.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.482+0000 7f60c2bcc7c0 10 moving default.rgw.meta+users.keys+JCZHIUSFADNDMNFEHUYO to cache LRU end 2024-01-16T04:50:54.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.482+0000 7f60c2bcc7c0 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.keys:JCZHIUSFADNDMNFEHUYO, ofs0, ns] 2024-01-16T04:50:54.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.483+0000 7f609f7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 111669149697 cookie 93892584513472 notifier 4199 bl.length()=186 2024-01-16T04:50:54.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.483+0000 7f609f7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+JCZHIUSFADNDMNFEHUYO info.flags=0x7 2024-01-16T04:50:54.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.483+0000 7f609f7fe700 10 rgw watcher librados: moving default.rgw.meta+users.keys+JCZHIUSFADNDMNFEHUYO to cache LRU end 2024-01-16T04:50:54.481 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-01-16T04:50:54.481 INFO:teuthology.orchestra.run.smithi156.stdout: "user_id": "foo.client.0", 2024-01-16T04:50:54.481 INFO:teuthology.orchestra.run.smithi156.stdout: "display_name": "Mr. foo.client.0", 2024-01-16T04:50:54.481 INFO:teuthology.orchestra.run.smithi156.stdout: "email": "foo.client.0+test@test.test", 2024-01-16T04:50:54.481 INFO:teuthology.orchestra.run.smithi156.stdout: "suspended": 0, 2024-01-16T04:50:54.481 INFO:teuthology.orchestra.run.smithi156.stdout: "max_buckets": 1000, 2024-01-16T04:50:54.481 INFO:teuthology.orchestra.run.smithi156.stdout: "subusers": [], 2024-01-16T04:50:54.481 INFO:teuthology.orchestra.run.smithi156.stdout: "keys": [ 2024-01-16T04:50:54.481 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-01-16T04:50:54.481 INFO:teuthology.orchestra.run.smithi156.stdout: "user": "foo.client.0", 2024-01-16T04:50:54.481 INFO:teuthology.orchestra.run.smithi156.stdout: "access_key": "JCZHIUSFADNDMNFEHUYO", 2024-01-16T04:50:54.481 INFO:teuthology.orchestra.run.smithi156.stdout: "secret_key": "JNplZ8RWGrssQX6YvS4uNwsAXKSsThpUN1Mt3ndCTy/ejw9Mhl+W1A==" 2024-01-16T04:50:54.481 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-01-16T04:50:54.482 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2024-01-16T04:50:54.482 INFO:teuthology.orchestra.run.smithi156.stdout: "swift_keys": [], 2024-01-16T04:50:54.482 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": [ 2024-01-16T04:50:54.482 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-01-16T04:50:54.482 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "user-policy", 2024-01-16T04:50:54.482 INFO:teuthology.orchestra.run.smithi156.stdout: "perm": "*" 2024-01-16T04:50:54.482 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-01-16T04:50:54.482 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2024-01-16T04:50:54.482 INFO:teuthology.orchestra.run.smithi156.stdout: "op_mask": "read, write, delete", 2024-01-16T04:50:54.482 INFO:teuthology.orchestra.run.smithi156.stdout: "default_placement": "", 2024-01-16T04:50:54.482 INFO:teuthology.orchestra.run.smithi156.stdout: "default_storage_class": "", 2024-01-16T04:50:54.482 INFO:teuthology.orchestra.run.smithi156.stdout: "placement_tags": [], 2024-01-16T04:50:54.482 INFO:teuthology.orchestra.run.smithi156.stdout: "bucket_quota": { 2024-01-16T04:50:54.482 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": false, 2024-01-16T04:50:54.483 INFO:teuthology.orchestra.run.smithi156.stdout: "check_on_raw": false, 2024-01-16T04:50:54.483 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size": -1, 2024-01-16T04:50:54.483 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size_kb": 0, 2024-01-16T04:50:54.483 INFO:teuthology.orchestra.run.smithi156.stdout: "max_objects": -1 2024-01-16T04:50:54.483 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-01-16T04:50:54.483 INFO:teuthology.orchestra.run.smithi156.stdout: "user_quota": { 2024-01-16T04:50:54.483 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": false, 2024-01-16T04:50:54.483 INFO:teuthology.orchestra.run.smithi156.stdout: "check_on_raw": false, 2024-01-16T04:50:54.483 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size": -1, 2024-01-16T04:50:54.483 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size_kb": 0, 2024-01-16T04:50:54.483 INFO:teuthology.orchestra.run.smithi156.stdout: "max_objects": -1 2024-01-16T04:50:54.483 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-01-16T04:50:54.483 INFO:teuthology.orchestra.run.smithi156.stdout: "temp_url_keys": [], 2024-01-16T04:50:54.483 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "rgw", 2024-01-16T04:50:54.483 INFO:teuthology.orchestra.run.smithi156.stdout: "mfa_ids": [] 2024-01-16T04:50:54.484 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-01-16T04:50:54.484 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:50:54.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.484+0000 7f60c2bcc7c0 20 remove_watcher() i=0 2024-01-16T04:50:54.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.484+0000 7f60c2bcc7c0 2 removed watcher, disabling cache 2024-01-16T04:50:54.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.485+0000 7f60c2bcc7c0 20 remove_watcher() i=1 2024-01-16T04:50:54.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.486+0000 7f60c2bcc7c0 20 remove_watcher() i=2 2024-01-16T04:50:54.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.487+0000 7f60c2bcc7c0 20 remove_watcher() i=3 2024-01-16T04:50:54.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.488+0000 7f60c2bcc7c0 20 remove_watcher() i=4 2024-01-16T04:50:54.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.489+0000 7f60c2bcc7c0 20 remove_watcher() i=5 2024-01-16T04:50:54.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.489+0000 7f60c2bcc7c0 20 remove_watcher() i=6 2024-01-16T04:50:54.488 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.490+0000 7f60c2bcc7c0 20 remove_watcher() i=7 2024-01-16T04:50:54.488 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.490+0000 7f607f7de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:50:54.492 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 9048749329 --totp-seed VMB47PZC7TJQZ7DZMGJ5WGO25T76652R6XYK3YASQXHCW6JPI7VST6PCQG7BFNUA --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-01-16T04:50:54.542 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:50:54.542 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:50:54.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.550+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.551+0000 7f79fe6c07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.551+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.551+0000 7f79fe6c07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.551+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.552+0000 7f79fe6c07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.552+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.552+0000 7f79fe6c07c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:54.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.559+0000 7f79fe6c07c0 20 RGWRados::pool_iterate: got zonegroup_info.92d3822f-5de2-431f-9d8f-b46ea51d44b6 2024-01-16T04:50:54.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.559+0000 7f79fe6c07c0 20 RGWRados::pool_iterate: got zone_info.2259a717-7b4f-4cc8-b097-8339a13202a0 2024-01-16T04:50:54.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.559+0000 7f79fe6c07c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:50:54.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.559+0000 7f79fe6c07c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:50:54.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.559+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.559+0000 7f79fe6c07c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:54.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.559+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.560+0000 7f79fe6c07c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:50:54.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.560+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.560+0000 7f79fe6c07c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:54.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.560+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.560+0000 7f79fe6c07c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:50:54.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.560+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.560+0000 7f79fe6c07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.560+0000 7f79fe6c07c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:50:54.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.560+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.560+0000 7f79fe6c07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.560+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.561+0000 7f79fe6c07c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:54.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.561+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.561+0000 7f79fe6c07c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:50:54.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.561+0000 7f79fe6c07c0 20 zonegroup default 2024-01-16T04:50:54.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.561+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.561+0000 7f79fe6c07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.561+0000 7f79fe6c07c0 10 Cannot find current period zone using local zone 2024-01-16T04:50:54.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.561+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.561+0000 7f79fe6c07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.561+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.562+0000 7f79fe6c07c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:54.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.562+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.562+0000 7f79fe6c07c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:50:54.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.562+0000 7f79fe6c07c0 20 zone default found 2024-01-16T04:50:54.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.562+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.562+0000 7f79fe6c07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.562+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.562+0000 7f79fe6c07c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:54.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.562+0000 7f79fe6c07c0 20 started sync module instance, tier type = 2024-01-16T04:50:54.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.562+0000 7f79fe6c07c0 20 started zone id=2259a717-7b4f-4cc8-b097-8339a13202a0 (name=default) with tier type = 2024-01-16T04:50:54.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.570+0000 7f79fe6c07c0 20 add_watcher() i=0 2024-01-16T04:50:54.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.570+0000 7f79fe6c07c0 20 add_watcher() i=1 2024-01-16T04:50:54.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.570+0000 7f79fe6c07c0 20 add_watcher() i=2 2024-01-16T04:50:54.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.570+0000 7f79fe6c07c0 20 add_watcher() i=3 2024-01-16T04:50:54.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.570+0000 7f79fe6c07c0 20 add_watcher() i=4 2024-01-16T04:50:54.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.570+0000 7f79fe6c07c0 20 add_watcher() i=5 2024-01-16T04:50:54.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.570+0000 7f79fe6c07c0 20 add_watcher() i=6 2024-01-16T04:50:54.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.571+0000 7f79fe6c07c0 20 add_watcher() i=7 2024-01-16T04:50:54.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.571+0000 7f79fe6c07c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:50:54.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.572+0000 7f79bb7de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:50:54.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.572+0000 7f79fe6c07c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:50:54.569 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.572+0000 7f79fe6c07c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:50:54.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.576+0000 7f79fe6c07c0 5 note: GC not initialized 2024-01-16T04:50:54.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.577+0000 7f792a7fc700 20 reqs_thread_entry: start 2024-01-16T04:50:54.575 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.577+0000 7f791bfff700 20 reqs_thread_entry: start 2024-01-16T04:50:54.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.577+0000 7f79fe6c07c0 20 init_complete bucket index max shards: 11 2024-01-16T04:50:54.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.577+0000 7f79fe6c07c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:50:54.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.577+0000 7f791a7fc700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:51:24 2024 2024-01-16T04:50:54.576 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:50:54.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.577+0000 7f7919ffb700 20 reqs_thread_entry: start 2024-01-16T04:50:54.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.577+0000 7f79fe6c07c0 20 get_system_obj_state: rctx=0x7ffed5ccf158 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55947b921d50 s->prefetch_data=0 2024-01-16T04:50:54.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.577+0000 7f79fe6c07c0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2024-01-16T04:50:54.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.578+0000 7f79fe6c07c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2024-01-16T04:50:54.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.578+0000 7f79fe6c07c0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:50:54.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.578+0000 7f79fe6c07c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:50:54.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.578+0000 7f79fe6c07c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:50:54.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.578+0000 7f79fe6c07c0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:50:54.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.578+0000 7f79fe6c07c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:54.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.578+0000 7f79fe6c07c0 20 rados_obj.operate() r=0 bl.length=455 2024-01-16T04:50:54.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.578+0000 7f79fe6c07c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x11 2024-01-16T04:50:54.577 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:54.578+0000 7f79fe6c07c0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:50:56.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.205+0000 7f79fe6c07c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-01-16T04:50:56.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.205+0000 7f79fe6c07c0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:50:56.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.205+0000 7f79fe6c07c0 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2024-01-16T04:50:56.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.205+0000 7f79e2ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084289 cookie 94096218006320 notifier 4205 bl.length()=654 2024-01-16T04:50:56.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.205+0000 7f79e2ffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2024-01-16T04:50:56.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.205+0000 7f79e2ffd700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:50:56.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.208+0000 7f79fe6c07c0 20 remove_watcher() i=0 2024-01-16T04:50:56.205 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.208+0000 7f79fe6c07c0 2 removed watcher, disabling cache 2024-01-16T04:50:56.206 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.209+0000 7f79fe6c07c0 20 remove_watcher() i=1 2024-01-16T04:50:56.207 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.210+0000 7f79fe6c07c0 20 remove_watcher() i=2 2024-01-16T04:50:56.208 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.211+0000 7f79fe6c07c0 20 remove_watcher() i=3 2024-01-16T04:50:56.209 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.212+0000 7f79fe6c07c0 20 remove_watcher() i=4 2024-01-16T04:50:56.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.213+0000 7f79fe6c07c0 20 remove_watcher() i=5 2024-01-16T04:50:56.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.214+0000 7f79fe6c07c0 20 remove_watcher() i=6 2024-01-16T04:50:56.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.215+0000 7f79fe6c07c0 20 remove_watcher() i=7 2024-01-16T04:50:56.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.215+0000 7f79bb7de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:50:56.218 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2024-01-16T04:50:56.218 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --access-key QNZSUOJCMRFICNYBWQGD --secret qbp9OsdX5c656jkA5SDNR2ARzxOc+cky/ndfAxxBZ/RTYckZYDQ8fA== --email bar.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2024-01-16T04:50:56.269 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:50:56.269 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:50:56.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.278+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.279+0000 7fb123c4a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.276 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.279+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.279+0000 7fb123c4a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.279+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.279+0000 7fb123c4a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.279+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.277 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.280+0000 7fb123c4a7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.287+0000 7fb123c4a7c0 20 RGWRados::pool_iterate: got zonegroup_info.92d3822f-5de2-431f-9d8f-b46ea51d44b6 2024-01-16T04:50:56.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.287+0000 7fb123c4a7c0 20 RGWRados::pool_iterate: got zone_info.2259a717-7b4f-4cc8-b097-8339a13202a0 2024-01-16T04:50:56.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.287+0000 7fb123c4a7c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:50:56.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.287+0000 7fb123c4a7c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:50:56.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.287+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.287+0000 7fb123c4a7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.287+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.287+0000 7fb123c4a7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:50:56.285 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.287+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.286 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.288+0000 7fb123c4a7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.286 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.288+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.288+0000 7fb123c4a7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:50:56.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.288+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.288+0000 7fb123c4a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.288+0000 7fb123c4a7c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:50:56.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.288+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.288+0000 7fb123c4a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.288+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.289+0000 7fb123c4a7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.288 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.289+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.289+0000 7fb123c4a7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:50:56.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.289+0000 7fb123c4a7c0 20 zonegroup default 2024-01-16T04:50:56.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.289+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.289+0000 7fb123c4a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.289+0000 7fb123c4a7c0 10 Cannot find current period zone using local zone 2024-01-16T04:50:56.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.289+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.289+0000 7fb123c4a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.289+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.289+0000 7fb123c4a7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.289+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.290+0000 7fb123c4a7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:50:56.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.290+0000 7fb123c4a7c0 20 zone default found 2024-01-16T04:50:56.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.290+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.290+0000 7fb123c4a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.290+0000 7fb123c4a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.290+0000 7fb123c4a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.290+0000 7fb123c4a7c0 20 started sync module instance, tier type = 2024-01-16T04:50:56.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.290+0000 7fb123c4a7c0 20 started zone id=2259a717-7b4f-4cc8-b097-8339a13202a0 (name=default) with tier type = 2024-01-16T04:50:56.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.298+0000 7fb123c4a7c0 20 add_watcher() i=0 2024-01-16T04:50:56.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.298+0000 7fb123c4a7c0 20 add_watcher() i=1 2024-01-16T04:50:56.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.298+0000 7fb123c4a7c0 20 add_watcher() i=2 2024-01-16T04:50:56.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.298+0000 7fb123c4a7c0 20 add_watcher() i=3 2024-01-16T04:50:56.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.298+0000 7fb123c4a7c0 20 add_watcher() i=4 2024-01-16T04:50:56.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.298+0000 7fb123c4a7c0 20 add_watcher() i=5 2024-01-16T04:50:56.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.298+0000 7fb123c4a7c0 20 add_watcher() i=6 2024-01-16T04:50:56.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.299+0000 7fb123c4a7c0 20 add_watcher() i=7 2024-01-16T04:50:56.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.299+0000 7fb123c4a7c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:50:56.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.300+0000 7fb05bfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:50:56.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.300+0000 7fb123c4a7c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:50:56.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.300+0000 7fb123c4a7c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:50:56.301 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.304+0000 7fb123c4a7c0 5 note: GC not initialized 2024-01-16T04:50:56.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.305+0000 7fb04affd700 20 reqs_thread_entry: start 2024-01-16T04:50:56.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.305+0000 7fb048ff9700 20 reqs_thread_entry: start 2024-01-16T04:50:56.304 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.305+0000 7fb123c4a7c0 20 init_complete bucket index max shards: 11 2024-01-16T04:50:56.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.306+0000 7fb123c4a7c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:50:56.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.306+0000 7fb042ffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:51:26 2024 2024-01-16T04:50:56.305 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:50:56.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.306+0000 7fb0427fc700 20 reqs_thread_entry: start 2024-01-16T04:50:56.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.306+0000 7fb123c4a7c0 20 get_system_obj_state: rctx=0x7ffe21df0408 obj=default.rgw.meta:users.uid:bar.client.0 state=0x560e56ff2d50 s->prefetch_data=0 2024-01-16T04:50:56.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.306+0000 7fb123c4a7c0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2024-01-16T04:50:56.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.306+0000 7fb123c4a7c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2024-01-16T04:50:56.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.306+0000 7fb123c4a7c0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:50:56.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.306+0000 7fb123c4a7c0 20 get_system_obj_state: rctx=0x7ffe21df0418 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x560e56ff2d50 s->prefetch_data=0 2024-01-16T04:50:56.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.306+0000 7fb123c4a7c0 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2024-01-16T04:50:56.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.306+0000 7fb123c4a7c0 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2024-01-16T04:50:56.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.306+0000 7fb123c4a7c0 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2024-01-16T04:50:56.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.306+0000 7fb123c4a7c0 20 get_system_obj_state: rctx=0x7ffe21df0418 obj=default.rgw.meta:users.keys:QNZSUOJCMRFICNYBWQGD state=0x560e56ff2d50 s->prefetch_data=0 2024-01-16T04:50:56.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.306+0000 7fb123c4a7c0 10 cache get: name=default.rgw.meta+users.keys+QNZSUOJCMRFICNYBWQGD : miss 2024-01-16T04:50:56.305 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.307+0000 7fb123c4a7c0 10 cache put: name=default.rgw.meta+users.keys+QNZSUOJCMRFICNYBWQGD info.flags=0x0 2024-01-16T04:50:56.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.307+0000 7fb123c4a7c0 10 adding default.rgw.meta+users.keys+QNZSUOJCMRFICNYBWQGD to cache LRU end 2024-01-16T04:50:56.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.307+0000 7fb123c4a7c0 20 get_system_obj_state: rctx=0x7ffe21defd38 obj=default.rgw.meta:users.keys:QNZSUOJCMRFICNYBWQGD state=0x560e56ff2d50 s->prefetch_data=0 2024-01-16T04:50:56.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.307+0000 7fb123c4a7c0 10 cache get: name=default.rgw.meta+users.keys+QNZSUOJCMRFICNYBWQGD : hit (negative entry) 2024-01-16T04:50:56.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.307+0000 7fb123c4a7c0 20 get_system_obj_state: rctx=0x7ffe21df0078 obj=default.rgw.meta:users.keys:QNZSUOJCMRFICNYBWQGD state=0x560e56ff2d50 s->prefetch_data=0 2024-01-16T04:50:56.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.307+0000 7fb123c4a7c0 10 cache get: name=default.rgw.meta+users.keys+QNZSUOJCMRFICNYBWQGD : hit (negative entry) 2024-01-16T04:50:56.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.308+0000 7fb123c4a7c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-01-16T04:50:56.306 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.308+0000 7fb123c4a7c0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:50:56.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.308+0000 7fb123c4a7c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2024-01-16T04:50:56.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.308+0000 7fb10cda6700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084290 cookie 94619590412048 notifier 4211 bl.length()=640 2024-01-16T04:50:56.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.308+0000 7fb10cda6700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-01-16T04:50:56.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.308+0000 7fb10cda6700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:50:56.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.309+0000 7fb123c4a7c0 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2024-01-16T04:50:56.307 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.309+0000 7fb123c4a7c0 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2024-01-16T04:50:56.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.309+0000 7fb123c4a7c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2024-01-16T04:50:56.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.310+0000 7fb10cda6700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084291 cookie 94619590412048 notifier 4211 bl.length()=194 2024-01-16T04:50:56.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.310+0000 7fb10cda6700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2024-01-16T04:50:56.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.310+0000 7fb10cda6700 10 rgw watcher librados: moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2024-01-16T04:50:56.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.311+0000 7fb123c4a7c0 10 cache put: name=default.rgw.meta+users.keys+QNZSUOJCMRFICNYBWQGD info.flags=0x7 2024-01-16T04:50:56.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.311+0000 7fb123c4a7c0 10 moving default.rgw.meta+users.keys+QNZSUOJCMRFICNYBWQGD to cache LRU end 2024-01-16T04:50:56.308 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.311+0000 7fb123c4a7c0 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.keys:QNZSUOJCMRFICNYBWQGD, ofs0, ns] 2024-01-16T04:50:56.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.311+0000 7fb10cda6700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084288 cookie 94619590417584 notifier 4211 bl.length()=186 2024-01-16T04:50:56.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.311+0000 7fb10cda6700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+QNZSUOJCMRFICNYBWQGD info.flags=0x7 2024-01-16T04:50:56.309 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.311+0000 7fb10cda6700 10 rgw watcher librados: moving default.rgw.meta+users.keys+QNZSUOJCMRFICNYBWQGD to cache LRU end 2024-01-16T04:50:56.309 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-01-16T04:50:56.310 INFO:teuthology.orchestra.run.smithi156.stdout: "user_id": "bar.client.0", 2024-01-16T04:50:56.310 INFO:teuthology.orchestra.run.smithi156.stdout: "display_name": "Mr. bar.client.0", 2024-01-16T04:50:56.310 INFO:teuthology.orchestra.run.smithi156.stdout: "email": "bar.client.0+test@test.test", 2024-01-16T04:50:56.310 INFO:teuthology.orchestra.run.smithi156.stdout: "suspended": 0, 2024-01-16T04:50:56.310 INFO:teuthology.orchestra.run.smithi156.stdout: "max_buckets": 1000, 2024-01-16T04:50:56.310 INFO:teuthology.orchestra.run.smithi156.stdout: "subusers": [], 2024-01-16T04:50:56.310 INFO:teuthology.orchestra.run.smithi156.stdout: "keys": [ 2024-01-16T04:50:56.310 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-01-16T04:50:56.310 INFO:teuthology.orchestra.run.smithi156.stdout: "user": "bar.client.0", 2024-01-16T04:50:56.310 INFO:teuthology.orchestra.run.smithi156.stdout: "access_key": "QNZSUOJCMRFICNYBWQGD", 2024-01-16T04:50:56.310 INFO:teuthology.orchestra.run.smithi156.stdout: "secret_key": "qbp9OsdX5c656jkA5SDNR2ARzxOc+cky/ndfAxxBZ/RTYckZYDQ8fA==" 2024-01-16T04:50:56.310 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-01-16T04:50:56.310 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2024-01-16T04:50:56.310 INFO:teuthology.orchestra.run.smithi156.stdout: "swift_keys": [], 2024-01-16T04:50:56.310 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": [ 2024-01-16T04:50:56.311 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-01-16T04:50:56.311 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "user-policy", 2024-01-16T04:50:56.311 INFO:teuthology.orchestra.run.smithi156.stdout: "perm": "*" 2024-01-16T04:50:56.311 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-01-16T04:50:56.311 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2024-01-16T04:50:56.311 INFO:teuthology.orchestra.run.smithi156.stdout: "op_mask": "read, write, delete", 2024-01-16T04:50:56.311 INFO:teuthology.orchestra.run.smithi156.stdout: "default_placement": "", 2024-01-16T04:50:56.311 INFO:teuthology.orchestra.run.smithi156.stdout: "default_storage_class": "", 2024-01-16T04:50:56.311 INFO:teuthology.orchestra.run.smithi156.stdout: "placement_tags": [], 2024-01-16T04:50:56.311 INFO:teuthology.orchestra.run.smithi156.stdout: "bucket_quota": { 2024-01-16T04:50:56.311 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": false, 2024-01-16T04:50:56.311 INFO:teuthology.orchestra.run.smithi156.stdout: "check_on_raw": false, 2024-01-16T04:50:56.311 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size": -1, 2024-01-16T04:50:56.311 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size_kb": 0, 2024-01-16T04:50:56.312 INFO:teuthology.orchestra.run.smithi156.stdout: "max_objects": -1 2024-01-16T04:50:56.312 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-01-16T04:50:56.312 INFO:teuthology.orchestra.run.smithi156.stdout: "user_quota": { 2024-01-16T04:50:56.312 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": false, 2024-01-16T04:50:56.312 INFO:teuthology.orchestra.run.smithi156.stdout: "check_on_raw": false, 2024-01-16T04:50:56.312 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size": -1, 2024-01-16T04:50:56.312 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size_kb": 0, 2024-01-16T04:50:56.312 INFO:teuthology.orchestra.run.smithi156.stdout: "max_objects": -1 2024-01-16T04:50:56.312 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-01-16T04:50:56.312 INFO:teuthology.orchestra.run.smithi156.stdout: "temp_url_keys": [], 2024-01-16T04:50:56.312 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "rgw", 2024-01-16T04:50:56.312 INFO:teuthology.orchestra.run.smithi156.stdout: "mfa_ids": [] 2024-01-16T04:50:56.312 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-01-16T04:50:56.313 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:50:56.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.313+0000 7fb123c4a7c0 20 remove_watcher() i=0 2024-01-16T04:50:56.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.313+0000 7fb123c4a7c0 2 removed watcher, disabling cache 2024-01-16T04:50:56.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.314+0000 7fb123c4a7c0 20 remove_watcher() i=1 2024-01-16T04:50:56.313 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.315+0000 7fb123c4a7c0 20 remove_watcher() i=2 2024-01-16T04:50:56.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.316+0000 7fb123c4a7c0 20 remove_watcher() i=3 2024-01-16T04:50:56.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.317+0000 7fb123c4a7c0 20 remove_watcher() i=4 2024-01-16T04:50:56.315 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.318+0000 7fb123c4a7c0 20 remove_watcher() i=5 2024-01-16T04:50:56.316 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.318+0000 7fb123c4a7c0 20 remove_watcher() i=6 2024-01-16T04:50:56.316 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.319+0000 7fb123c4a7c0 20 remove_watcher() i=7 2024-01-16T04:50:56.317 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.319+0000 7fb05bfff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:50:56.321 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 0951787533 --totp-seed 7KIEVQQPG6AI5UJS6M546CT6JL44RO2M3WPZEHPKEOZ7AUUTMCJSKNQKSDDIEBQC --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-01-16T04:50:56.370 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:50:56.370 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:50:56.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.379+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.380+0000 7f10508fb7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.380+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.380+0000 7f10508fb7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.380+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.380+0000 7f10508fb7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.380+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.381+0000 7f10508fb7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.388+0000 7f10508fb7c0 20 RGWRados::pool_iterate: got zonegroup_info.92d3822f-5de2-431f-9d8f-b46ea51d44b6 2024-01-16T04:50:56.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.388+0000 7f10508fb7c0 20 RGWRados::pool_iterate: got zone_info.2259a717-7b4f-4cc8-b097-8339a13202a0 2024-01-16T04:50:56.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.388+0000 7f10508fb7c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:50:56.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.388+0000 7f10508fb7c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:50:56.387 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.388+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.388+0000 7f10508fb7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.388+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.388+0000 7f10508fb7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:50:56.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.388+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.388+0000 7f10508fb7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.388+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.389+0000 7f10508fb7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:50:56.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.389+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.389+0000 7f10508fb7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.389+0000 7f10508fb7c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:50:56.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.389+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.389+0000 7f10508fb7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.389+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.388 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.389+0000 7f10508fb7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.389+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.389+0000 7f10508fb7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:50:56.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.389+0000 7f10508fb7c0 20 zonegroup default 2024-01-16T04:50:56.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.389+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.390+0000 7f10508fb7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.390+0000 7f10508fb7c0 10 Cannot find current period zone using local zone 2024-01-16T04:50:56.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.390+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.390+0000 7f10508fb7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.390+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.390+0000 7f10508fb7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.389 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.390+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.390+0000 7f10508fb7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:50:56.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.390+0000 7f10508fb7c0 20 zone default found 2024-01-16T04:50:56.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.390+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.390 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.391+0000 7f10508fb7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.391 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.391+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.391 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.391+0000 7f10508fb7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.391 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.391+0000 7f10508fb7c0 20 started sync module instance, tier type = 2024-01-16T04:50:56.391 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.391+0000 7f10508fb7c0 20 started zone id=2259a717-7b4f-4cc8-b097-8339a13202a0 (name=default) with tier type = 2024-01-16T04:50:56.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.398+0000 7f10508fb7c0 20 add_watcher() i=0 2024-01-16T04:50:56.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.398+0000 7f10508fb7c0 20 add_watcher() i=1 2024-01-16T04:50:56.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.398+0000 7f10508fb7c0 20 add_watcher() i=2 2024-01-16T04:50:56.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.398+0000 7f10508fb7c0 20 add_watcher() i=3 2024-01-16T04:50:56.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.398+0000 7f10508fb7c0 20 add_watcher() i=4 2024-01-16T04:50:56.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.398+0000 7f10508fb7c0 20 add_watcher() i=5 2024-01-16T04:50:56.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.398+0000 7f10508fb7c0 20 add_watcher() i=6 2024-01-16T04:50:56.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.399+0000 7f10508fb7c0 20 add_watcher() i=7 2024-01-16T04:50:56.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.399+0000 7f10508fb7c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:50:56.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.400+0000 7f100d7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:50:56.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.400+0000 7f10508fb7c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:50:56.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.400+0000 7f10508fb7c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:50:56.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.404+0000 7f10508fb7c0 5 note: GC not initialized 2024-01-16T04:50:56.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.405+0000 7f0f7bfff700 20 reqs_thread_entry: start 2024-01-16T04:50:56.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.405+0000 7f0f79ffb700 20 reqs_thread_entry: start 2024-01-16T04:50:56.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.405+0000 7f10508fb7c0 20 init_complete bucket index max shards: 11 2024-01-16T04:50:56.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.405+0000 7f10508fb7c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:50:56.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.405+0000 7f0f67fff700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:51:26 2024 2024-01-16T04:50:56.403 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:50:56.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.405+0000 7f0f677fe700 20 reqs_thread_entry: start 2024-01-16T04:50:56.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.405+0000 7f10508fb7c0 20 get_system_obj_state: rctx=0x7ffc9e585ef8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x55cb66056d50 s->prefetch_data=0 2024-01-16T04:50:56.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.405+0000 7f10508fb7c0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2024-01-16T04:50:56.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.406+0000 7f10508fb7c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2024-01-16T04:50:56.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.406+0000 7f10508fb7c0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:50:56.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.406+0000 7f10508fb7c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:50:56.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.406+0000 7f10508fb7c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:50:56.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.406+0000 7f10508fb7c0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:50:56.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.406+0000 7f10508fb7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.405 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.406+0000 7f10508fb7c0 20 rados_obj.operate() r=0 bl.length=455 2024-01-16T04:50:56.405 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.406+0000 7f10508fb7c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x11 2024-01-16T04:50:56.405 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.406+0000 7f10508fb7c0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:50:56.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.409+0000 7f10508fb7c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-01-16T04:50:56.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.409+0000 7f10508fb7c0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:50:56.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.409+0000 7f10508fb7c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2024-01-16T04:50:56.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.409+0000 7f1039a57700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084292 cookie 94332079665840 notifier 4217 bl.length()=654 2024-01-16T04:50:56.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.409+0000 7f1039a57700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2024-01-16T04:50:56.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.409+0000 7f1039a57700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:50:56.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.411+0000 7f10508fb7c0 20 remove_watcher() i=0 2024-01-16T04:50:56.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.411+0000 7f10508fb7c0 2 removed watcher, disabling cache 2024-01-16T04:50:56.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.412+0000 7f10508fb7c0 20 remove_watcher() i=1 2024-01-16T04:50:56.410 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.413+0000 7f10508fb7c0 20 remove_watcher() i=2 2024-01-16T04:50:56.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.413+0000 7f10508fb7c0 20 remove_watcher() i=3 2024-01-16T04:50:56.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.414+0000 7f10508fb7c0 20 remove_watcher() i=4 2024-01-16T04:50:56.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.415+0000 7f10508fb7c0 20 remove_watcher() i=5 2024-01-16T04:50:56.413 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.416+0000 7f10508fb7c0 20 remove_watcher() i=6 2024-01-16T04:50:56.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.417+0000 7f10508fb7c0 20 remove_watcher() i=7 2024-01-16T04:50:56.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.417+0000 7f100d7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:50:56.419 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2024-01-16T04:50:56.419 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --access-key WFOUTFGFFAJQEWTWXRIU --secret HDy9hDKxsxRdHeE6Yq7e4Umsq150ou8ddqDuY2zFSOXX5qv2UvR0Qg== --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --cluster ceph 2024-01-16T04:50:56.468 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:50:56.468 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:50:56.474 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.476+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.477+0000 7f352c68b7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.477+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.477+0000 7f352c68b7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.477+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.478+0000 7f352c68b7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.478+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.478+0000 7f352c68b7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.485+0000 7f352c68b7c0 20 RGWRados::pool_iterate: got zonegroup_info.92d3822f-5de2-431f-9d8f-b46ea51d44b6 2024-01-16T04:50:56.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.485+0000 7f352c68b7c0 20 RGWRados::pool_iterate: got zone_info.2259a717-7b4f-4cc8-b097-8339a13202a0 2024-01-16T04:50:56.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.485+0000 7f352c68b7c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:50:56.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.485+0000 7f352c68b7c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:50:56.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.485+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.485+0000 7f352c68b7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.485+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.485+0000 7f352c68b7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:50:56.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.485+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.485+0000 7f352c68b7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.485+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.486+0000 7f352c68b7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:50:56.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.486+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.486+0000 7f352c68b7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.486+0000 7f352c68b7c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:50:56.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.486+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.486+0000 7f352c68b7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.486+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.486+0000 7f352c68b7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.486+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.486+0000 7f352c68b7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:50:56.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.486+0000 7f352c68b7c0 20 zonegroup default 2024-01-16T04:50:56.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.486+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.487+0000 7f352c68b7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.487+0000 7f352c68b7c0 10 Cannot find current period zone using local zone 2024-01-16T04:50:56.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.487+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.487+0000 7f352c68b7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.487+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.487+0000 7f352c68b7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.487+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.487+0000 7f352c68b7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:50:56.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.487+0000 7f352c68b7c0 20 zone default found 2024-01-16T04:50:56.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.487+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.488+0000 7f352c68b7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.488 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.488+0000 7f352c68b7c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.488 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.488+0000 7f352c68b7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.488 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.488+0000 7f352c68b7c0 20 started sync module instance, tier type = 2024-01-16T04:50:56.488 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.488+0000 7f352c68b7c0 20 started zone id=2259a717-7b4f-4cc8-b097-8339a13202a0 (name=default) with tier type = 2024-01-16T04:50:56.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.495+0000 7f352c68b7c0 20 add_watcher() i=0 2024-01-16T04:50:56.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.495+0000 7f352c68b7c0 20 add_watcher() i=1 2024-01-16T04:50:56.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.495+0000 7f352c68b7c0 20 add_watcher() i=2 2024-01-16T04:50:56.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.495+0000 7f352c68b7c0 20 add_watcher() i=3 2024-01-16T04:50:56.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.495+0000 7f352c68b7c0 20 add_watcher() i=4 2024-01-16T04:50:56.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.495+0000 7f352c68b7c0 20 add_watcher() i=5 2024-01-16T04:50:56.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.495+0000 7f352c68b7c0 20 add_watcher() i=6 2024-01-16T04:50:56.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.496+0000 7f352c68b7c0 20 add_watcher() i=7 2024-01-16T04:50:56.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.496+0000 7f352c68b7c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:50:56.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.497+0000 7f3463fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:50:56.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.497+0000 7f352c68b7c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:50:56.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.497+0000 7f352c68b7c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:50:56.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.501+0000 7f352c68b7c0 5 note: GC not initialized 2024-01-16T04:50:56.501 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.502+0000 7f3452ffd700 20 reqs_thread_entry: start 2024-01-16T04:50:56.501 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.502+0000 7f34517fa700 20 reqs_thread_entry: start 2024-01-16T04:50:56.501 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.502+0000 7f352c68b7c0 20 init_complete bucket index max shards: 11 2024-01-16T04:50:56.501 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.502+0000 7f352c68b7c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:50:56.501 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.502+0000 7f344affd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:51:26 2024 2024-01-16T04:50:56.501 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:50:56.501 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f344a7fc700 20 reqs_thread_entry: start 2024-01-16T04:50:56.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f352c68b7c0 20 get_system_obj_state: rctx=0x7fff0d6d2bd8 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x562768de7d50 s->prefetch_data=0 2024-01-16T04:50:56.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f352c68b7c0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2024-01-16T04:50:56.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f352c68b7c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2024-01-16T04:50:56.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f352c68b7c0 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:50:56.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f352c68b7c0 20 get_system_obj_state: rctx=0x7fff0d6d2be8 obj=default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test state=0x562768de7d50 s->prefetch_data=0 2024-01-16T04:50:56.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f352c68b7c0 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2024-01-16T04:50:56.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f352c68b7c0 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2024-01-16T04:50:56.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f352c68b7c0 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2024-01-16T04:50:56.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f352c68b7c0 20 get_system_obj_state: rctx=0x7fff0d6d2be8 obj=default.rgw.meta:users.keys:WFOUTFGFFAJQEWTWXRIU state=0x562768de7d50 s->prefetch_data=0 2024-01-16T04:50:56.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f352c68b7c0 10 cache get: name=default.rgw.meta+users.keys+WFOUTFGFFAJQEWTWXRIU : miss 2024-01-16T04:50:56.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f352c68b7c0 10 cache put: name=default.rgw.meta+users.keys+WFOUTFGFFAJQEWTWXRIU info.flags=0x0 2024-01-16T04:50:56.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f352c68b7c0 10 adding default.rgw.meta+users.keys+WFOUTFGFFAJQEWTWXRIU to cache LRU end 2024-01-16T04:50:56.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f352c68b7c0 20 get_system_obj_state: rctx=0x7fff0d6d2508 obj=default.rgw.meta:users.keys:WFOUTFGFFAJQEWTWXRIU state=0x562768de7d50 s->prefetch_data=0 2024-01-16T04:50:56.503 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f352c68b7c0 10 cache get: name=default.rgw.meta+users.keys+WFOUTFGFFAJQEWTWXRIU : hit (negative entry) 2024-01-16T04:50:56.503 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f352c68b7c0 20 get_system_obj_state: rctx=0x7fff0d6d2848 obj=default.rgw.meta:users.keys:WFOUTFGFFAJQEWTWXRIU state=0x562768de7d50 s->prefetch_data=0 2024-01-16T04:50:56.503 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.503+0000 7f352c68b7c0 10 cache get: name=default.rgw.meta+users.keys+WFOUTFGFFAJQEWTWXRIU : hit (negative entry) 2024-01-16T04:50:56.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.505+0000 7f352c68b7c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-01-16T04:50:56.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.505+0000 7f352c68b7c0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:50:56.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.505+0000 7f352c68b7c0 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2024-01-16T04:50:56.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.505+0000 7f35157e7700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084290 cookie 94727264460400 notifier 4223 bl.length()=714 2024-01-16T04:50:56.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.505+0000 7f35157e7700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-01-16T04:50:56.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.505+0000 7f35157e7700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:50:56.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.506+0000 7f352c68b7c0 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2024-01-16T04:50:56.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.506+0000 7f352c68b7c0 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2024-01-16T04:50:56.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.506+0000 7f352c68b7c0 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2024-01-16T04:50:56.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.507+0000 7f35157e7700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084291 cookie 94727264466352 notifier 4223 bl.length()=224 2024-01-16T04:50:56.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.507+0000 7f35157e7700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2024-01-16T04:50:56.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.507+0000 7f35157e7700 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2024-01-16T04:50:56.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.508+0000 7f352c68b7c0 10 cache put: name=default.rgw.meta+users.keys+WFOUTFGFFAJQEWTWXRIU info.flags=0x7 2024-01-16T04:50:56.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.508+0000 7f352c68b7c0 10 moving default.rgw.meta+users.keys+WFOUTFGFFAJQEWTWXRIU to cache LRU end 2024-01-16T04:50:56.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.508+0000 7f352c68b7c0 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.keys:WFOUTFGFFAJQEWTWXRIU, ofs0, ns] 2024-01-16T04:50:56.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.508+0000 7f3514fe6700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084289 cookie 94727264449648 notifier 4223 bl.length()=201 2024-01-16T04:50:56.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.508+0000 7f3514fe6700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+WFOUTFGFFAJQEWTWXRIU info.flags=0x7 2024-01-16T04:50:56.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.508+0000 7f3514fe6700 10 rgw watcher librados: moving default.rgw.meta+users.keys+WFOUTFGFFAJQEWTWXRIU to cache LRU end 2024-01-16T04:50:56.506 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-01-16T04:50:56.506 INFO:teuthology.orchestra.run.smithi156.stdout: "user_id": "testx$tenanteduser.client.0", 2024-01-16T04:50:56.506 INFO:teuthology.orchestra.run.smithi156.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2024-01-16T04:50:56.506 INFO:teuthology.orchestra.run.smithi156.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2024-01-16T04:50:56.507 INFO:teuthology.orchestra.run.smithi156.stdout: "suspended": 0, 2024-01-16T04:50:56.507 INFO:teuthology.orchestra.run.smithi156.stdout: "max_buckets": 1000, 2024-01-16T04:50:56.507 INFO:teuthology.orchestra.run.smithi156.stdout: "subusers": [], 2024-01-16T04:50:56.507 INFO:teuthology.orchestra.run.smithi156.stdout: "keys": [ 2024-01-16T04:50:56.507 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-01-16T04:50:56.507 INFO:teuthology.orchestra.run.smithi156.stdout: "user": "testx$tenanteduser.client.0", 2024-01-16T04:50:56.507 INFO:teuthology.orchestra.run.smithi156.stdout: "access_key": "WFOUTFGFFAJQEWTWXRIU", 2024-01-16T04:50:56.507 INFO:teuthology.orchestra.run.smithi156.stdout: "secret_key": "HDy9hDKxsxRdHeE6Yq7e4Umsq150ou8ddqDuY2zFSOXX5qv2UvR0Qg==" 2024-01-16T04:50:56.507 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-01-16T04:50:56.507 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2024-01-16T04:50:56.507 INFO:teuthology.orchestra.run.smithi156.stdout: "swift_keys": [], 2024-01-16T04:50:56.507 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": [ 2024-01-16T04:50:56.507 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-01-16T04:50:56.507 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "user-policy", 2024-01-16T04:50:56.508 INFO:teuthology.orchestra.run.smithi156.stdout: "perm": "*" 2024-01-16T04:50:56.508 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-01-16T04:50:56.508 INFO:teuthology.orchestra.run.smithi156.stdout: ], 2024-01-16T04:50:56.508 INFO:teuthology.orchestra.run.smithi156.stdout: "op_mask": "read, write, delete", 2024-01-16T04:50:56.508 INFO:teuthology.orchestra.run.smithi156.stdout: "default_placement": "", 2024-01-16T04:50:56.508 INFO:teuthology.orchestra.run.smithi156.stdout: "default_storage_class": "", 2024-01-16T04:50:56.508 INFO:teuthology.orchestra.run.smithi156.stdout: "placement_tags": [], 2024-01-16T04:50:56.508 INFO:teuthology.orchestra.run.smithi156.stdout: "bucket_quota": { 2024-01-16T04:50:56.508 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": false, 2024-01-16T04:50:56.508 INFO:teuthology.orchestra.run.smithi156.stdout: "check_on_raw": false, 2024-01-16T04:50:56.508 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size": -1, 2024-01-16T04:50:56.508 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size_kb": 0, 2024-01-16T04:50:56.508 INFO:teuthology.orchestra.run.smithi156.stdout: "max_objects": -1 2024-01-16T04:50:56.508 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-01-16T04:50:56.509 INFO:teuthology.orchestra.run.smithi156.stdout: "user_quota": { 2024-01-16T04:50:56.509 INFO:teuthology.orchestra.run.smithi156.stdout: "enabled": false, 2024-01-16T04:50:56.509 INFO:teuthology.orchestra.run.smithi156.stdout: "check_on_raw": false, 2024-01-16T04:50:56.509 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size": -1, 2024-01-16T04:50:56.509 INFO:teuthology.orchestra.run.smithi156.stdout: "max_size_kb": 0, 2024-01-16T04:50:56.509 INFO:teuthology.orchestra.run.smithi156.stdout: "max_objects": -1 2024-01-16T04:50:56.509 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-01-16T04:50:56.509 INFO:teuthology.orchestra.run.smithi156.stdout: "temp_url_keys": [], 2024-01-16T04:50:56.509 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "rgw", 2024-01-16T04:50:56.509 INFO:teuthology.orchestra.run.smithi156.stdout: "mfa_ids": [] 2024-01-16T04:50:56.509 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-01-16T04:50:56.509 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:50:56.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.510+0000 7f352c68b7c0 20 remove_watcher() i=0 2024-01-16T04:50:56.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.510+0000 7f352c68b7c0 2 removed watcher, disabling cache 2024-01-16T04:50:56.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.511+0000 7f352c68b7c0 20 remove_watcher() i=1 2024-01-16T04:50:56.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.512+0000 7f352c68b7c0 20 remove_watcher() i=2 2024-01-16T04:50:56.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.513+0000 7f352c68b7c0 20 remove_watcher() i=3 2024-01-16T04:50:56.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.514+0000 7f352c68b7c0 20 remove_watcher() i=4 2024-01-16T04:50:56.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.514+0000 7f352c68b7c0 20 remove_watcher() i=5 2024-01-16T04:50:56.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.515+0000 7f352c68b7c0 20 remove_watcher() i=6 2024-01-16T04:50:56.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.516+0000 7f352c68b7c0 20 remove_watcher() i=7 2024-01-16T04:50:56.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.516+0000 7f3463fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:50:56.518 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 8427456586 --totp-seed 2EZIN3T4DXNH3RSB44YFHGRKSRVJZBMECOI2VCMX6UEPYVQLG563W2YAEKRBZGFT --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2024-01-16T04:50:56.566 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:50:56.566 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:50:56.572 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.575+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.575+0000 7fae112077c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.575+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.576+0000 7fae112077c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.576+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.576+0000 7fae112077c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.576+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.576+0000 7fae112077c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.583+0000 7fae112077c0 20 RGWRados::pool_iterate: got zonegroup_info.92d3822f-5de2-431f-9d8f-b46ea51d44b6 2024-01-16T04:50:56.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.583+0000 7fae112077c0 20 RGWRados::pool_iterate: got zone_info.2259a717-7b4f-4cc8-b097-8339a13202a0 2024-01-16T04:50:56.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.583+0000 7fae112077c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:50:56.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.583+0000 7fae112077c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:50:56.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.583+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.583+0000 7fae112077c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.583+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.584+0000 7fae112077c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:50:56.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.584+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.584+0000 7fae112077c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.584+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.584+0000 7fae112077c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:50:56.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.584+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.584+0000 7fae112077c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.584+0000 7fae112077c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:50:56.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.584+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.584+0000 7fae112077c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.584+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.585+0000 7fae112077c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.585+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.585+0000 7fae112077c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:50:56.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.585+0000 7fae112077c0 20 zonegroup default 2024-01-16T04:50:56.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.585+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.585+0000 7fae112077c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.585+0000 7fae112077c0 10 Cannot find current period zone using local zone 2024-01-16T04:50:56.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.585+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.585+0000 7fae112077c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.585+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.586+0000 7fae112077c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:50:56.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.586+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.586+0000 7fae112077c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:50:56.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.586+0000 7fae112077c0 20 zone default found 2024-01-16T04:50:56.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.586+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.586+0000 7fae112077c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.586+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.586+0000 7fae112077c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:50:56.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.586+0000 7fae112077c0 20 started sync module instance, tier type = 2024-01-16T04:50:56.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.586+0000 7fae112077c0 20 started zone id=2259a717-7b4f-4cc8-b097-8339a13202a0 (name=default) with tier type = 2024-01-16T04:50:56.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.594+0000 7fae112077c0 20 add_watcher() i=0 2024-01-16T04:50:56.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.594+0000 7fae112077c0 20 add_watcher() i=1 2024-01-16T04:50:56.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.594+0000 7fae112077c0 20 add_watcher() i=2 2024-01-16T04:50:56.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.594+0000 7fae112077c0 20 add_watcher() i=3 2024-01-16T04:50:56.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.594+0000 7fae112077c0 20 add_watcher() i=4 2024-01-16T04:50:56.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.594+0000 7fae112077c0 20 add_watcher() i=5 2024-01-16T04:50:56.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.594+0000 7fae112077c0 20 add_watcher() i=6 2024-01-16T04:50:56.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.595+0000 7fae112077c0 20 add_watcher() i=7 2024-01-16T04:50:56.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.595+0000 7fae112077c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:50:56.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.596+0000 7fadcdfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:50:56.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.596+0000 7fae112077c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:50:56.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.596+0000 7fae112077c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:50:56.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.600+0000 7fae112077c0 5 note: GC not initialized 2024-01-16T04:50:56.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.601+0000 7fad44ff9700 20 reqs_thread_entry: start 2024-01-16T04:50:56.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.601+0000 7fad3a7fc700 20 reqs_thread_entry: start 2024-01-16T04:50:56.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.601+0000 7fae112077c0 20 init_complete bucket index max shards: 11 2024-01-16T04:50:56.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.601+0000 7fae112077c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:50:56.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.601+0000 7fad38ff9700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:51:26 2024 2024-01-16T04:50:56.599 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:50:56.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.601+0000 7fad1ffff700 20 reqs_thread_entry: start 2024-01-16T04:50:56.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.601+0000 7fae112077c0 20 get_system_obj_state: rctx=0x7ffdd35451f8 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x559e49302d50 s->prefetch_data=0 2024-01-16T04:50:56.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.601+0000 7fae112077c0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2024-01-16T04:50:56.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.602+0000 7fae112077c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2024-01-16T04:50:56.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.602+0000 7fae112077c0 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:50:56.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.602+0000 7fae112077c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:50:56.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.602+0000 7fae112077c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:50:56.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.602+0000 7fae112077c0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:50:56.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.602+0000 7fae112077c0 20 rados->read ofs=0 len=0 2024-01-16T04:50:56.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.602+0000 7fae112077c0 20 rados_obj.operate() r=0 bl.length=514 2024-01-16T04:50:56.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.602+0000 7fae112077c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x11 2024-01-16T04:50:56.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.602+0000 7fae112077c0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:50:56.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.605+0000 7fae112077c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-01-16T04:50:56.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.605+0000 7fae112077c0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:50:56.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.605+0000 7fae112077c0 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2024-01-16T04:50:56.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.605+0000 7fadfa363700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084292 cookie 94138322418448 notifier 4229 bl.length()=728 2024-01-16T04:50:56.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.605+0000 7fadfa363700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2024-01-16T04:50:56.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.605+0000 7fadfa363700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:50:56.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.607+0000 7fae112077c0 20 remove_watcher() i=0 2024-01-16T04:50:56.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.607+0000 7fae112077c0 2 removed watcher, disabling cache 2024-01-16T04:50:56.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.608+0000 7fae112077c0 20 remove_watcher() i=1 2024-01-16T04:50:56.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.608+0000 7fae112077c0 20 remove_watcher() i=2 2024-01-16T04:50:56.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.609+0000 7fae112077c0 20 remove_watcher() i=3 2024-01-16T04:50:56.607 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.610+0000 7fae112077c0 20 remove_watcher() i=4 2024-01-16T04:50:56.608 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.611+0000 7fae112077c0 20 remove_watcher() i=5 2024-01-16T04:50:56.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.611+0000 7fae112077c0 20 remove_watcher() i=6 2024-01-16T04:50:56.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.612+0000 7fae112077c0 20 remove_watcher() i=7 2024-01-16T04:50:56.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:50:56.612+0000 7fadcdfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:50:56.614 INFO:tasks.s3tests:Configuring s3-tests... 2024-01-16T04:50:56.615 DEBUG:teuthology.orchestra.run.smithi156:> cd /home/ubuntu/cephtest/s3-tests && ./bootstrap 2024-01-16T04:51:05.726 INFO:teuthology.orchestra.run.smithi156.stderr:./bootstrap: Missing required RPM packages: libevent-devel libffi-devel libxml2-devel libxslt-devel zlib-devel. 2024-01-16T04:51:06.576 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:15 ago on Tue 16 Jan 2024 04:46:51 AM UTC. 2024-01-16T04:51:08.299 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-01-16T04:51:08.301 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:51:08.301 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-01-16T04:51:08.301 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:51:08.301 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-01-16T04:51:08.301 INFO:teuthology.orchestra.run.smithi156.stdout: libevent-devel x86_64 2.1.8-5.el8 appstream 104 k 2024-01-16T04:51:08.301 INFO:teuthology.orchestra.run.smithi156.stdout: libffi-devel x86_64 3.1-24.el8 baseos 29 k 2024-01-16T04:51:08.301 INFO:teuthology.orchestra.run.smithi156.stdout: libxml2-devel x86_64 2.9.7-18.el8 appstream 1.1 M 2024-01-16T04:51:08.302 INFO:teuthology.orchestra.run.smithi156.stdout: libxslt-devel x86_64 1.1.32-6.el8 appstream 323 k 2024-01-16T04:51:08.302 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel x86_64 1.2.11-25.el8 baseos 59 k 2024-01-16T04:51:08.302 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2024-01-16T04:51:08.302 INFO:teuthology.orchestra.run.smithi156.stdout: libxml2 x86_64 2.9.7-18.el8 baseos 697 k 2024-01-16T04:51:08.302 INFO:teuthology.orchestra.run.smithi156.stdout: python3-libxml2 x86_64 2.9.7-18.el8 baseos 238 k 2024-01-16T04:51:08.302 INFO:teuthology.orchestra.run.smithi156.stdout: zlib x86_64 1.2.11-25.el8 baseos 103 k 2024-01-16T04:51:08.302 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-01-16T04:51:08.302 INFO:teuthology.orchestra.run.smithi156.stdout: cmake-filesystem x86_64 3.26.5-2.el8 appstream 52 k 2024-01-16T04:51:08.302 INFO:teuthology.orchestra.run.smithi156.stdout: libgcrypt-devel x86_64 1.8.5-7.el8 baseos 150 k 2024-01-16T04:51:08.302 INFO:teuthology.orchestra.run.smithi156.stdout: libgpg-error-devel x86_64 1.31-1.el8 baseos 64 k 2024-01-16T04:51:08.302 INFO:teuthology.orchestra.run.smithi156.stdout: xz-devel x86_64 5.2.4-4.el8 baseos 63 k 2024-01-16T04:51:08.302 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:08.302 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-01-16T04:51:08.303 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-01-16T04:51:08.303 INFO:teuthology.orchestra.run.smithi156.stdout:Install 9 Packages 2024-01-16T04:51:08.303 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 3 Packages 2024-01-16T04:51:08.303 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:08.303 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 2.9 M 2024-01-16T04:51:08.303 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-01-16T04:51:08.663 INFO:teuthology.orchestra.run.smithi156.stdout:(1/12): cmake-filesystem-3.26.5-2.el8.x86_64.rp 388 kB/s | 52 kB 00:00 2024-01-16T04:51:08.688 INFO:teuthology.orchestra.run.smithi156.stdout:(2/12): libevent-devel-2.1.8-5.el8.x86_64.rpm 655 kB/s | 104 kB 00:00 2024-01-16T04:51:08.747 INFO:teuthology.orchestra.run.smithi156.stdout:(3/12): libffi-devel-3.1-24.el8.x86_64.rpm 498 kB/s | 29 kB 00:00 2024-01-16T04:51:08.781 INFO:teuthology.orchestra.run.smithi156.stdout:(4/12): libxslt-devel-1.1.32-6.el8.x86_64.rpm 2.7 MB/s | 323 kB 00:00 2024-01-16T04:51:08.814 INFO:teuthology.orchestra.run.smithi156.stdout:(5/12): libxml2-devel-2.9.7-18.el8.x86_64.rpm 3.9 MB/s | 1.1 MB 00:00 2024-01-16T04:51:08.840 INFO:teuthology.orchestra.run.smithi156.stdout:(6/12): libgcrypt-devel-1.8.5-7.el8.x86_64.rpm 1.6 MB/s | 150 kB 00:00 2024-01-16T04:51:08.874 INFO:teuthology.orchestra.run.smithi156.stdout:(7/12): libgpg-error-devel-1.31-1.el8.x86_64.rp 692 kB/s | 64 kB 00:00 2024-01-16T04:51:08.890 INFO:teuthology.orchestra.run.smithi156.stdout:(8/12): zlib-devel-1.2.11-25.el8.x86_64.rpm 1.1 MB/s | 59 kB 00:00 2024-01-16T04:51:08.935 INFO:teuthology.orchestra.run.smithi156.stdout:(9/12): xz-devel-5.2.4-4.el8.x86_64.rpm 520 kB/s | 63 kB 00:00 2024-01-16T04:51:08.974 INFO:teuthology.orchestra.run.smithi156.stdout:(10/12): libxml2-2.9.7-18.el8.x86_64.rpm 6.8 MB/s | 697 kB 00:00 2024-01-16T04:51:08.999 INFO:teuthology.orchestra.run.smithi156.stdout:(11/12): python3-libxml2-2.9.7-18.el8.x86_64.rp 2.1 MB/s | 238 kB 00:00 2024-01-16T04:51:09.025 INFO:teuthology.orchestra.run.smithi156.stdout:(12/12): zlib-1.2.11-25.el8.x86_64.rpm 1.1 MB/s | 103 kB 00:00 2024-01-16T04:51:09.025 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-01-16T04:51:09.025 INFO:teuthology.orchestra.run.smithi156.stdout:Total 4.1 MB/s | 2.9 MB 00:00 2024-01-16T04:51:09.141 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-01-16T04:51:09.280 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-01-16T04:51:09.280 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-01-16T04:51:09.430 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-01-16T04:51:09.433 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-01-16T04:51:09.938 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-01-16T04:51:10.215 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : zlib-1.2.11-25.el8.x86_64 1/15 2024-01-16T04:51:10.501 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libxml2-2.9.7-18.el8.x86_64 2/15 2024-01-16T04:51:10.695 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgpg-error-devel-1.31-1.el8.x86_64 3/15 2024-01-16T04:51:10.718 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libgpg-error-devel-1.31-1.el8.x86_64 3/15 2024-01-16T04:51:10.978 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgcrypt-devel-1.8.5-7.el8.x86_64 4/15 2024-01-16T04:51:10.991 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libgcrypt-devel-1.8.5-7.el8.x86_64 4/15 2024-01-16T04:51:11.174 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : zlib-devel-1.2.11-25.el8.x86_64 5/15 2024-01-16T04:51:11.352 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xz-devel-5.2.4-4.el8.x86_64 6/15 2024-01-16T04:51:11.809 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cmake-filesystem-3.26.5-2.el8.x86_64 7/15 2024-01-16T04:51:12.145 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libxml2-devel-2.9.7-18.el8.x86_64 8/15 2024-01-16T04:51:12.413 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libxslt-devel-1.1.32-6.el8.x86_64 9/15 2024-01-16T04:51:12.683 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : python3-libxml2-2.9.7-18.el8.x86_64 10/15 2024-01-16T04:51:12.819 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libffi-devel-3.1-24.el8.x86_64 11/15 2024-01-16T04:51:12.848 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libffi-devel-3.1-24.el8.x86_64 11/15 2024-01-16T04:51:13.070 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libevent-devel-2.1.8-5.el8.x86_64 12/15 2024-01-16T04:51:13.254 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : python3-libxml2-2.9.7-16.el8.x86_64 13/15 2024-01-16T04:51:13.453 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libxml2-2.9.7-16.el8.x86_64 14/15 2024-01-16T04:51:13.644 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : zlib-1.2.11-21.el8.x86_64 15/15 2024-01-16T04:51:13.998 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: zlib-1.2.11-21.el8.x86_64 15/15 2024-01-16T04:51:13.998 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cmake-filesystem-3.26.5-2.el8.x86_64 1/15 2024-01-16T04:51:13.998 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libevent-devel-2.1.8-5.el8.x86_64 2/15 2024-01-16T04:51:13.998 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libxml2-devel-2.9.7-18.el8.x86_64 3/15 2024-01-16T04:51:13.998 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libxslt-devel-1.1.32-6.el8.x86_64 4/15 2024-01-16T04:51:13.998 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libffi-devel-3.1-24.el8.x86_64 5/15 2024-01-16T04:51:13.998 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgcrypt-devel-1.8.5-7.el8.x86_64 6/15 2024-01-16T04:51:13.999 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgpg-error-devel-1.31-1.el8.x86_64 7/15 2024-01-16T04:51:13.999 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xz-devel-5.2.4-4.el8.x86_64 8/15 2024-01-16T04:51:13.999 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : zlib-devel-1.2.11-25.el8.x86_64 9/15 2024-01-16T04:51:13.999 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libxml2-2.9.7-18.el8.x86_64 10/15 2024-01-16T04:51:13.999 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libxml2-2.9.7-16.el8.x86_64 11/15 2024-01-16T04:51:13.999 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-libxml2-2.9.7-18.el8.x86_64 12/15 2024-01-16T04:51:13.999 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-libxml2-2.9.7-16.el8.x86_64 13/15 2024-01-16T04:51:13.999 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : zlib-1.2.11-25.el8.x86_64 14/15 2024-01-16T04:51:14.901 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : zlib-1.2.11-21.el8.x86_64 15/15 2024-01-16T04:51:14.901 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:14.902 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2024-01-16T04:51:14.902 INFO:teuthology.orchestra.run.smithi156.stdout: libxml2-2.9.7-18.el8.x86_64 python3-libxml2-2.9.7-18.el8.x86_64 2024-01-16T04:51:14.902 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-1.2.11-25.el8.x86_64 2024-01-16T04:51:14.902 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-01-16T04:51:14.902 INFO:teuthology.orchestra.run.smithi156.stdout: cmake-filesystem-3.26.5-2.el8.x86_64 libevent-devel-2.1.8-5.el8.x86_64 2024-01-16T04:51:14.902 INFO:teuthology.orchestra.run.smithi156.stdout: libffi-devel-3.1-24.el8.x86_64 libgcrypt-devel-1.8.5-7.el8.x86_64 2024-01-16T04:51:14.902 INFO:teuthology.orchestra.run.smithi156.stdout: libgpg-error-devel-1.31-1.el8.x86_64 libxml2-devel-2.9.7-18.el8.x86_64 2024-01-16T04:51:14.902 INFO:teuthology.orchestra.run.smithi156.stdout: libxslt-devel-1.1.32-6.el8.x86_64 xz-devel-5.2.4-4.el8.x86_64 2024-01-16T04:51:14.902 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel-1.2.11-25.el8.x86_64 2024-01-16T04:51:14.902 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:14.902 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-01-16T04:51:15.802 INFO:teuthology.orchestra.run.smithi156.stdout:Already using interpreter /usr/bin/python3.6 2024-01-16T04:51:15.802 INFO:teuthology.orchestra.run.smithi156.stdout:Using base prefix '/usr' 2024-01-16T04:51:15.802 INFO:teuthology.orchestra.run.smithi156.stdout:New python executable in /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python3.6 2024-01-16T04:51:15.802 INFO:teuthology.orchestra.run.smithi156.stdout:Also creating executable in /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python 2024-01-16T04:51:19.995 INFO:teuthology.orchestra.run.smithi156.stdout:Installing setuptools, pip, wheel...done. 2024-01-16T04:51:20.335 INFO:teuthology.orchestra.run.smithi156.stdout:Requirement already satisfied: pip in ./virtualenv/lib/python3.6/site-packages (21.3.1) 2024-01-16T04:51:21.692 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting setuptools==44.1.0 2024-01-16T04:51:21.793 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB) 2024-01-16T04:51:21.998 INFO:teuthology.orchestra.run.smithi156.stdout:Installing collected packages: setuptools 2024-01-16T04:51:21.998 INFO:teuthology.orchestra.run.smithi156.stdout: Attempting uninstall: setuptools 2024-01-16T04:51:21.998 INFO:teuthology.orchestra.run.smithi156.stdout: Found existing installation: setuptools 59.6.0 2024-01-16T04:51:22.043 INFO:teuthology.orchestra.run.smithi156.stdout: Uninstalling setuptools-59.6.0: 2024-01-16T04:51:22.052 INFO:teuthology.orchestra.run.smithi156.stdout: Successfully uninstalled setuptools-59.6.0 2024-01-16T04:51:22.283 INFO:teuthology.orchestra.run.smithi156.stdout:Successfully installed setuptools-44.1.0 2024-01-16T04:51:22.896 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting PyYAML 2024-01-16T04:51:22.973 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB) 2024-01-16T04:51:23.148 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting nose>=1.0.0 2024-01-16T04:51:23.168 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading nose-1.3.7-py3-none-any.whl (154 kB) 2024-01-16T04:51:23.286 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting boto>=2.6.0 2024-01-16T04:51:23.307 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2024-01-16T04:51:24.354 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting boto3>=1.0.0 2024-01-16T04:51:24.372 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading boto3-1.23.10-py3-none-any.whl (132 kB) 2024-01-16T04:51:25.508 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting botocore<1.28.0 2024-01-16T04:51:25.530 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading botocore-1.26.10-py3-none-any.whl (8.8 MB) 2024-01-16T04:51:25.885 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting munch>=2.0.0 2024-01-16T04:51:25.913 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading munch-4.0.0-py2.py3-none-any.whl (9.9 kB) 2024-01-16T04:51:26.375 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting gevent>=1.0 2024-01-16T04:51:26.395 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading gevent-22.10.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.8 MB) 2024-01-16T04:51:26.619 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting isodate>=0.4.4 2024-01-16T04:51:26.636 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB) 2024-01-16T04:51:26.759 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting requests>=2.23.0 2024-01-16T04:51:26.777 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-01-16T04:51:26.962 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting pytz>=2011k 2024-01-16T04:51:26.983 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading pytz-2023.3.post1-py2.py3-none-any.whl (502 kB) 2024-01-16T04:51:27.101 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting httplib2 2024-01-16T04:51:27.140 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading httplib2-0.22.0-py3-none-any.whl (96 kB) 2024-01-16T04:51:27.799 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting lxml 2024-01-16T04:51:27.819 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading lxml-5.1.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.1 MB) 2024-01-16T04:51:28.281 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting s3transfer<0.6.0,>=0.5.0 2024-01-16T04:51:28.300 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2024-01-16T04:51:28.371 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting jmespath<2.0.0,>=0.7.1 2024-01-16T04:51:28.389 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2024-01-16T04:51:28.533 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting urllib3<1.27,>=1.25.4 2024-01-16T04:51:28.552 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2024-01-16T04:51:28.647 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting python-dateutil<3.0.0,>=2.1 2024-01-16T04:51:28.667 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2024-01-16T04:51:28.875 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting importlib-metadata>=1.7.0 2024-01-16T04:51:28.893 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading importlib_metadata-4.8.3-py3-none-any.whl (17 kB) 2024-01-16T04:51:29.046 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting zope.event 2024-01-16T04:51:29.065 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading zope.event-4.6-py2.py3-none-any.whl (6.8 kB) 2024-01-16T04:51:29.093 INFO:teuthology.orchestra.run.smithi156.stdout:Requirement already satisfied: setuptools in ./virtualenv/lib/python3.6/site-packages (from gevent>=1.0->-r requirements.txt (line 9)) (44.1.0) 2024-01-16T04:51:29.594 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting greenlet>=2.0.0 2024-01-16T04:51:29.615 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading greenlet-2.0.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (564 kB) 2024-01-16T04:51:30.063 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting zope.interface 2024-01-16T04:51:30.083 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading zope.interface-5.5.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (253 kB) 2024-01-16T04:51:30.189 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting six 2024-01-16T04:51:30.206 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2024-01-16T04:51:30.601 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting charset-normalizer~=2.0.0 2024-01-16T04:51:30.619 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-01-16T04:51:30.738 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting certifi>=2017.4.17 2024-01-16T04:51:30.762 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading certifi-2023.11.17-py3-none-any.whl (162 kB) 2024-01-16T04:51:30.842 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting idna<4,>=2.5 2024-01-16T04:51:30.859 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading idna-3.6-py3-none-any.whl (61 kB) 2024-01-16T04:51:31.001 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 2024-01-16T04:51:31.019 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB) 2024-01-16T04:51:31.151 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting typing-extensions>=3.6.4 2024-01-16T04:51:31.171 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading typing_extensions-4.1.1-py3-none-any.whl (26 kB) 2024-01-16T04:51:31.279 INFO:teuthology.orchestra.run.smithi156.stdout:Collecting zipp>=0.5 2024-01-16T04:51:31.332 INFO:teuthology.orchestra.run.smithi156.stdout: Downloading zipp-3.6.0-py3-none-any.whl (5.3 kB) 2024-01-16T04:51:31.821 INFO:teuthology.orchestra.run.smithi156.stdout:Installing collected packages: six, urllib3, python-dateutil, jmespath, zipp, typing-extensions, botocore, zope.interface, zope.event, s3transfer, pyparsing, importlib-metadata, idna, greenlet, charset-normalizer, certifi, requests, PyYAML, pytz, nose, munch, lxml, isodate, httplib2, gevent, boto3, boto 2024-01-16T04:51:35.521 INFO:teuthology.orchestra.run.smithi156.stdout:Successfully installed PyYAML-6.0.1 boto-2.49.0 boto3-1.23.10 botocore-1.26.10 certifi-2023.11.17 charset-normalizer-2.0.12 gevent-22.10.2 greenlet-2.0.2 httplib2-0.22.0 idna-3.6 importlib-metadata-4.8.3 isodate-0.6.1 jmespath-0.10.0 lxml-5.1.0 munch-4.0.0 nose-1.3.7 pyparsing-3.1.1 python-dateutil-2.8.2 pytz-2023.3.post1 requests-2.27.1 s3transfer-0.5.2 six-1.16.0 typing-extensions-4.1.1 urllib3-1.26.18 zipp-3.6.0 zope.event-4.6 zope.interface-5.5.2 2024-01-16T04:51:35.840 INFO:teuthology.orchestra.run.smithi156.stdout:running develop 2024-01-16T04:51:35.861 INFO:teuthology.orchestra.run.smithi156.stdout:running egg_info 2024-01-16T04:51:35.862 INFO:teuthology.orchestra.run.smithi156.stdout:creating s3tests.egg-info 2024-01-16T04:51:35.862 INFO:teuthology.orchestra.run.smithi156.stdout:writing s3tests.egg-info/PKG-INFO 2024-01-16T04:51:35.862 INFO:teuthology.orchestra.run.smithi156.stdout:writing dependency_links to s3tests.egg-info/dependency_links.txt 2024-01-16T04:51:35.862 INFO:teuthology.orchestra.run.smithi156.stdout:writing requirements to s3tests.egg-info/requires.txt 2024-01-16T04:51:35.862 INFO:teuthology.orchestra.run.smithi156.stdout:writing top-level names to s3tests.egg-info/top_level.txt 2024-01-16T04:51:35.863 INFO:teuthology.orchestra.run.smithi156.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2024-01-16T04:51:35.872 INFO:teuthology.orchestra.run.smithi156.stdout:reading manifest file 's3tests.egg-info/SOURCES.txt' 2024-01-16T04:51:35.872 INFO:teuthology.orchestra.run.smithi156.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2024-01-16T04:51:35.884 INFO:teuthology.orchestra.run.smithi156.stdout:running build_ext 2024-01-16T04:51:35.885 INFO:teuthology.orchestra.run.smithi156.stdout:Creating /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/s3tests.egg-link (link to .) 2024-01-16T04:51:35.885 INFO:teuthology.orchestra.run.smithi156.stdout:Adding s3tests 0.0.1 to easy-install.pth file 2024-01-16T04:51:35.885 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:35.885 INFO:teuthology.orchestra.run.smithi156.stdout:Installed /home/ubuntu/cephtest/s3-tests 2024-01-16T04:51:35.886 INFO:teuthology.orchestra.run.smithi156.stdout:Processing dependencies for s3tests==0.0.1 2024-01-16T04:51:35.995 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for isodate==0.6.1 2024-01-16T04:51:35.996 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: isodate 0.6.1 2024-01-16T04:51:35.996 INFO:teuthology.orchestra.run.smithi156.stdout:Adding isodate 0.6.1 to easy-install.pth file 2024-01-16T04:51:35.996 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:35.996 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:35.997 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for gevent==22.10.2 2024-01-16T04:51:35.997 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: gevent 22.10.2 2024-01-16T04:51:35.997 INFO:teuthology.orchestra.run.smithi156.stdout:Adding gevent 22.10.2 to easy-install.pth file 2024-01-16T04:51:35.997 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:35.997 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:35.997 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for munch==4.0.0 2024-01-16T04:51:35.997 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: munch 4.0.0 2024-01-16T04:51:35.997 INFO:teuthology.orchestra.run.smithi156.stdout:Adding munch 4.0.0 to easy-install.pth file 2024-01-16T04:51:35.997 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:35.998 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:35.999 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for PyYAML==6.0.1 2024-01-16T04:51:35.999 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: PyYAML 6.0.1 2024-01-16T04:51:35.999 INFO:teuthology.orchestra.run.smithi156.stdout:Adding PyYAML 6.0.1 to easy-install.pth file 2024-01-16T04:51:35.999 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:35.999 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:35.999 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for boto3==1.23.10 2024-01-16T04:51:35.999 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: boto3 1.23.10 2024-01-16T04:51:35.999 INFO:teuthology.orchestra.run.smithi156.stdout:Adding boto3 1.23.10 to easy-install.pth file 2024-01-16T04:51:35.999 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:35.999 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:36.000 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for boto==2.49.0 2024-01-16T04:51:36.000 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: boto 2.49.0 2024-01-16T04:51:36.000 INFO:teuthology.orchestra.run.smithi156.stdout:Adding boto 2.49.0 to easy-install.pth file 2024-01-16T04:51:36.000 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:36.000 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:36.001 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for six==1.16.0 2024-01-16T04:51:36.001 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: six 1.16.0 2024-01-16T04:51:36.001 INFO:teuthology.orchestra.run.smithi156.stdout:Adding six 1.16.0 to easy-install.pth file 2024-01-16T04:51:36.001 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:36.001 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:36.002 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for zope.interface==5.5.2 2024-01-16T04:51:36.002 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: zope.interface 5.5.2 2024-01-16T04:51:36.002 INFO:teuthology.orchestra.run.smithi156.stdout:Adding zope.interface 5.5.2 to easy-install.pth file 2024-01-16T04:51:36.002 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:36.002 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:36.002 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for setuptools==44.1.0 2024-01-16T04:51:36.002 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: setuptools 44.1.0 2024-01-16T04:51:36.002 INFO:teuthology.orchestra.run.smithi156.stdout:Adding setuptools 44.1.0 to easy-install.pth file 2024-01-16T04:51:36.003 INFO:teuthology.orchestra.run.smithi156.stdout:Installing easy_install script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2024-01-16T04:51:36.003 INFO:teuthology.orchestra.run.smithi156.stdout:Installing easy_install-3.8 script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2024-01-16T04:51:36.003 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:36.004 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:36.005 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for zope.event==4.6 2024-01-16T04:51:36.005 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: zope.event 4.6 2024-01-16T04:51:36.005 INFO:teuthology.orchestra.run.smithi156.stdout:Adding zope.event 4.6 to easy-install.pth file 2024-01-16T04:51:36.005 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:36.005 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:36.005 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for greenlet==2.0.2 2024-01-16T04:51:36.005 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: greenlet 2.0.2 2024-01-16T04:51:36.005 INFO:teuthology.orchestra.run.smithi156.stdout:Adding greenlet 2.0.2 to easy-install.pth file 2024-01-16T04:51:36.005 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:36.005 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:36.006 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for importlib-metadata==4.8.3 2024-01-16T04:51:36.006 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: importlib-metadata 4.8.3 2024-01-16T04:51:36.006 INFO:teuthology.orchestra.run.smithi156.stdout:Adding importlib-metadata 4.8.3 to easy-install.pth file 2024-01-16T04:51:36.006 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:36.007 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:36.007 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for botocore==1.26.10 2024-01-16T04:51:36.007 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: botocore 1.26.10 2024-01-16T04:51:36.007 INFO:teuthology.orchestra.run.smithi156.stdout:Adding botocore 1.26.10 to easy-install.pth file 2024-01-16T04:51:36.007 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:36.007 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:36.008 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for jmespath==0.10.0 2024-01-16T04:51:36.008 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: jmespath 0.10.0 2024-01-16T04:51:36.008 INFO:teuthology.orchestra.run.smithi156.stdout:Adding jmespath 0.10.0 to easy-install.pth file 2024-01-16T04:51:36.008 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:36.008 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:36.008 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for s3transfer==0.5.2 2024-01-16T04:51:36.008 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: s3transfer 0.5.2 2024-01-16T04:51:36.008 INFO:teuthology.orchestra.run.smithi156.stdout:Adding s3transfer 0.5.2 to easy-install.pth file 2024-01-16T04:51:36.008 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:36.009 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:36.009 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for zipp==3.6.0 2024-01-16T04:51:36.010 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: zipp 3.6.0 2024-01-16T04:51:36.010 INFO:teuthology.orchestra.run.smithi156.stdout:Adding zipp 3.6.0 to easy-install.pth file 2024-01-16T04:51:36.010 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:36.010 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:36.010 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for typing-extensions==4.1.1 2024-01-16T04:51:36.010 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: typing-extensions 4.1.1 2024-01-16T04:51:36.010 INFO:teuthology.orchestra.run.smithi156.stdout:Adding typing-extensions 4.1.1 to easy-install.pth file 2024-01-16T04:51:36.010 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:36.010 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:36.011 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for python-dateutil==2.8.2 2024-01-16T04:51:36.011 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: python-dateutil 2.8.2 2024-01-16T04:51:36.011 INFO:teuthology.orchestra.run.smithi156.stdout:Adding python-dateutil 2.8.2 to easy-install.pth file 2024-01-16T04:51:36.011 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:36.012 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:36.012 INFO:teuthology.orchestra.run.smithi156.stdout:Searching for urllib3==1.26.18 2024-01-16T04:51:36.012 INFO:teuthology.orchestra.run.smithi156.stdout:Best match: urllib3 1.26.18 2024-01-16T04:51:36.012 INFO:teuthology.orchestra.run.smithi156.stdout:Adding urllib3 1.26.18 to easy-install.pth file 2024-01-16T04:51:36.012 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:51:36.012 INFO:teuthology.orchestra.run.smithi156.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages 2024-01-16T04:51:36.012 INFO:teuthology.orchestra.run.smithi156.stdout:Finished processing dependencies for s3tests==0.0.1 2024-01-16T04:51:36.030 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:51:36.030 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2024-01-16T04:51:36.048 INFO:tasks.s3tests:Configuring boto... 2024-01-16T04:51:36.048 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-01-16T04:51:36.049 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/boto.cfg 2024-01-16T04:51:36.109 DEBUG:teuthology.orchestra.run.smithi156:s3 tests against rgw> S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain' 2024-01-16T04:51:36.560 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok 2024-01-16T04:51:36.610 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok 2024-01-16T04:51:36.661 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_date_and_amz_date ... ok 2024-01-16T04:51:36.712 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok 2024-01-16T04:51:36.764 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_none ... ok 2024-01-16T04:51:36.773 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_contentlength_none ... ok 2024-01-16T04:51:36.826 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_acl_create_contentlength_none ... ok 2024-01-16T04:51:36.836 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok 2024-01-16T04:51:36.877 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_empty ... ok 2024-01-16T04:51:36.919 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_none ... ok 2024-01-16T04:51:36.971 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws2 ... ok 2024-01-16T04:51:37.021 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws2 ... ok 2024-01-16T04:51:37.072 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws2 ... ok 2024-01-16T04:51:37.122 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws2 ... ok 2024-01-16T04:51:37.123 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws2 ... ok 2024-01-16T04:51:37.169 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws2 ... ok 2024-01-16T04:51:37.169 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws4 ... SKIP 2024-01-16T04:51:37.169 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws4 ... SKIP 2024-01-16T04:51:37.170 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws4 ... SKIP 2024-01-16T04:51:37.170 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws4 ... SKIP 2024-01-16T04:51:37.170 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_empty_aws4 ... SKIP 2024-01-16T04:51:37.170 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_none_aws4 ... SKIP 2024-01-16T04:51:37.170 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_date_invalid_aws4 ... SKIP 2024-01-16T04:51:37.170 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_invalid_aws4 ... SKIP 2024-01-16T04:51:37.170 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_date_empty_aws4 ... SKIP 2024-01-16T04:51:37.170 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_empty_aws4 ... SKIP 2024-01-16T04:51:37.170 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws4 ... SKIP 2024-01-16T04:51:37.170 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_none_aws4 ... SKIP 2024-01-16T04:51:37.171 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_today_aws4 ... SKIP 2024-01-16T04:51:37.171 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_today_aws4 ... SKIP 2024-01-16T04:51:37.171 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_today_aws4 ... SKIP 2024-01-16T04:51:37.171 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_today_aws4 ... SKIP 2024-01-16T04:51:37.171 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_epoch_aws4 ... SKIP 2024-01-16T04:51:37.171 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_epoch_aws4 ... SKIP 2024-01-16T04:51:37.171 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_end_aws4 ... SKIP 2024-01-16T04:51:37.171 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_end_aws4 ... SKIP 2024-01-16T04:51:37.171 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_custom_header_aws4 ... SKIP 2024-01-16T04:51:37.171 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_header_aws4 ... SKIP 2024-01-16T04:51:37.171 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws4 ... SKIP 2024-01-16T04:51:37.171 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_empty_aws4 ... SKIP 2024-01-16T04:51:37.171 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_none_aws4 ... SKIP 2024-01-16T04:51:37.172 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_invalid_aws4 ... SKIP 2024-01-16T04:51:37.172 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_invalid_aws4 ... SKIP 2024-01-16T04:51:37.172 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_empty_aws4 ... SKIP 2024-01-16T04:51:37.172 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_empty_aws4 ... SKIP 2024-01-16T04:51:37.172 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws4 ... SKIP 2024-01-16T04:51:37.172 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_none_aws4 ... SKIP 2024-01-16T04:51:37.172 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_today_aws4 ... SKIP 2024-01-16T04:51:37.172 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_today_aws4 ... SKIP 2024-01-16T04:51:37.172 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_after_today_aws4 ... SKIP 2024-01-16T04:51:37.172 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_after_today_aws4 ... SKIP 2024-01-16T04:51:37.172 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws4 ... SKIP 2024-01-16T04:51:37.172 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_epoch_aws4 ... SKIP 2024-01-16T04:51:37.310 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_versioning_obj_read_not_exist_null ... ok 2024-01-16T04:51:37.310 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_lifecycle_transition ... SKIP 2024-01-16T04:51:37.310 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_lifecycle_transition_single_rule_multi_trans ... SKIP 2024-01-16T04:51:37.310 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_lifecycle_set_noncurrent_transition ... SKIP 2024-01-16T04:51:37.310 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_object_storage_class ... SKIP 2024-01-16T04:51:37.310 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_object_storage_class_multipart ... SKIP 2024-01-16T04:51:37.311 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_object_modify_storage_class ... SKIP 2024-01-16T04:51:37.311 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_object_modify_storage_class_multipart ... SKIP 2024-01-16T04:51:37.311 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_object_storage_class_copy ... SKIP 2024-01-16T04:51:37.311 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_object_storage_class_copy_multipart ... SKIP 2024-01-16T04:51:37.399 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_atomic_dual_conditional_write_1mb ... ok 2024-01-16T04:51:37.438 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_atomic_write_bucket_gone ... ok 2024-01-16T04:51:39.312 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_1 ... ok 2024-01-16T04:51:39.399 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_2 ... ok 2024-01-16T04:51:39.465 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_bucket_policy_set_condition_operator_end_with_IfExists ... ok 2024-01-16T04:51:39.524 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_enc ... ok 2024-01-16T04:51:39.541 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_request_obj_tag ... ok 2024-01-16T04:51:39.620 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.check_can_test_website ... SKIP 2024-01-16T04:51:39.620 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_nonexistant_bucket_rgw ... SKIP 2024-01-16T04:51:39.620 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_public_index ... SKIP 2024-01-16T04:51:39.620 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_public_index ... SKIP 2024-01-16T04:51:39.621 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty ... SKIP 2024-01-16T04:51:39.621 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty ... SKIP 2024-01-16T04:51:39.621 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index ... SKIP 2024-01-16T04:51:39.621 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index ... SKIP 2024-01-16T04:51:39.621 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_missingerrordoc ... SKIP 2024-01-16T04:51:39.621 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_missingerrordoc ... SKIP 2024-01-16T04:51:39.621 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_missingerrordoc ... SKIP 2024-01-16T04:51:39.621 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_missingerrordoc ... SKIP 2024-01-16T04:51:39.621 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_blockederrordoc ... SKIP 2024-01-16T04:51:39.621 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_pubilc_errordoc ... SKIP 2024-01-16T04:51:39.621 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_blockederrordoc ... SKIP 2024-01-16T04:51:39.621 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_blockederrordoc ... SKIP 2024-01-16T04:51:39.621 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_blockederrordoc ... SKIP 2024-01-16T04:51:39.622 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_gooderrordoc ... SKIP 2024-01-16T04:51:39.622 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_gooderrordoc ... SKIP 2024-01-16T04:51:39.622 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_gooderrordoc ... SKIP 2024-01-16T04:51:39.622 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_gooderrordoc ... SKIP 2024-01-16T04:51:39.622 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_base ... SKIP 2024-01-16T04:51:39.622 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path ... SKIP 2024-01-16T04:51:39.622 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path_upgrade ... SKIP 2024-01-16T04:51:39.622 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_xredirect_nonwebsite ... SKIP 2024-01-16T04:51:39.622 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_relative ... SKIP 2024-01-16T04:51:39.622 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_abs ... SKIP 2024-01-16T04:51:39.622 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_relative ... SKIP 2024-01-16T04:51:39.623 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_abs ... SKIP 2024-01-16T04:51:39.623 INFO:teuthology.orchestra.run.smithi156.stderr:SKIP 2024-01-16T04:51:43.124 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests.functional.test_utils.test_generate ... ok 2024-01-16T04:51:43.124 INFO:teuthology.orchestra.run.smithi156.stderr:/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2024-01-16T04:51:43.124 INFO:teuthology.orchestra.run.smithi156.stderr: warnings.warn(warning, PythonDeprecationWarning) 2024-01-16T04:51:43.148 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok 2024-01-16T04:51:50.912 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_bad ... ok 2024-01-16T04:51:50.941 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_empty ... ok 2024-01-16T04:51:50.984 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_none ... ok 2024-01-16T04:51:51.023 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_mismatch ... ok 2024-01-16T04:51:51.062 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_empty ... ok 2024-01-16T04:51:51.100 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_none ... ok 2024-01-16T04:51:51.147 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contentlength_negative ... ok 2024-01-16T04:51:51.185 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok 2024-01-16T04:51:51.211 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_empty ... ok 2024-01-16T04:51:51.237 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_none ... ok 2024-01-16T04:51:51.261 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_bucket_put_bad_canned_acl ... ok 2024-01-16T04:51:51.276 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_mismatch ... ok 2024-01-16T04:51:51.291 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_empty ... ok 2024-01-16T04:51:51.300 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_contentlength_negative ... ok 2024-01-16T04:51:51.324 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws2 ... ok 2024-01-16T04:51:51.355 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_empty_aws2 ... ok 2024-01-16T04:51:51.386 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_none_aws2 ... ok 2024-01-16T04:51:51.410 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_invalid_aws2 ... ok 2024-01-16T04:51:51.433 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_empty_aws2 ... ok 2024-01-16T04:51:51.458 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_today_aws2 ... ok 2024-01-16T04:51:51.481 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_epoch_aws2 ... ok 2024-01-16T04:51:51.503 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_after_end_aws2 ... ok 2024-01-16T04:51:51.519 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_empty_aws2 ... ok 2024-01-16T04:51:51.534 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_none_aws2 ... ok 2024-01-16T04:51:51.542 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_invalid_aws2 ... ok 2024-01-16T04:51:51.550 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_empty_aws2 ... ok 2024-01-16T04:51:51.559 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_today_aws2 ... ok 2024-01-16T04:51:51.566 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_after_today_aws2 ... ok 2024-01-16T04:51:51.573 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws2 ... ok 2024-01-16T04:51:51.651 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_empty ... ok 2024-01-16T04:51:51.740 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_distinct ... ok 2024-01-16T04:51:51.832 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_many ... ok 2024-01-16T04:51:51.923 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_many ... ok 2024-01-16T04:51:52.010 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_basic_key_count ... ok 2024-01-16T04:51:52.102 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_basic ... ok 2024-01-16T04:51:52.143 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_basic ... ok 2024-01-16T04:51:52.199 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_encoding_basic ... ok 2024-01-16T04:51:52.253 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_encoding_basic ... ok 2024-01-16T04:51:52.373 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix ... ok 2024-01-16T04:51:52.492 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix ... ok 2024-01-16T04:51:52.525 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_ends_with_delimiter ... ok 2024-01-16T04:51:52.560 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_ends_with_delimiter ... ok 2024-01-16T04:51:52.614 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_alt ... ok 2024-01-16T04:51:52.667 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_alt ... ok 2024-01-16T04:51:52.787 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_underscore ... ok 2024-01-16T04:51:52.932 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_underscore ... ok 2024-01-16T04:51:52.987 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_percentage ... ok 2024-01-16T04:51:53.040 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_percentage ... ok 2024-01-16T04:51:53.093 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_whitespace ... ok 2024-01-16T04:51:53.147 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_whitespace ... ok 2024-01-16T04:51:53.201 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_dot ... ok 2024-01-16T04:51:53.256 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_dot ... ok 2024-01-16T04:51:53.310 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_unreadable ... ok 2024-01-16T04:51:53.365 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_unreadable ... ok 2024-01-16T04:51:53.420 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_empty ... ok 2024-01-16T04:51:53.475 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_empty ... ok 2024-01-16T04:51:53.527 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_none ... ok 2024-01-16T04:51:53.582 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_none ... ok 2024-01-16T04:51:53.657 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_notempty ... ok 2024-01-16T04:51:53.704 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_defaultempty ... ok 2024-01-16T04:51:53.751 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_empty ... ok 2024-01-16T04:51:53.806 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_exist ... ok 2024-01-16T04:51:53.860 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_not_exist ... ok 2024-01-16T04:52:00.623 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_skip_special ... ok 2024-01-16T04:52:00.697 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_basic ... ok 2024-01-16T04:52:00.744 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_basic ... ok 2024-01-16T04:52:00.792 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_alt ... ok 2024-01-16T04:52:00.840 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_alt ... ok 2024-01-16T04:52:00.887 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_empty ... ok 2024-01-16T04:52:00.934 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_empty ... ok 2024-01-16T04:52:00.983 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_none ... ok 2024-01-16T04:52:01.030 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_none ... ok 2024-01-16T04:52:01.077 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_not_exist ... ok 2024-01-16T04:52:01.124 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_not_exist ... ok 2024-01-16T04:52:01.170 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_unreadable ... ok 2024-01-16T04:52:01.218 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_unreadable ... ok 2024-01-16T04:52:01.272 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_basic ... ok 2024-01-16T04:52:01.325 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_basic ... ok 2024-01-16T04:52:01.379 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_alt ... ok 2024-01-16T04:52:01.434 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_alt ... ok 2024-01-16T04:52:01.514 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_not_exist ... ok 2024-01-16T04:52:01.567 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_not_exist ... ok 2024-01-16T04:52:01.620 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_delimiter_not_exist ... ok 2024-01-16T04:52:01.675 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_delimiter_not_exist ... ok 2024-01-16T04:52:01.729 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_delimiter_not_exist ... ok 2024-01-16T04:52:01.782 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist ... ok 2024-01-16T04:52:01.882 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_one ... ok 2024-01-16T04:52:01.982 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_one ... ok 2024-01-16T04:52:02.036 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_zero ... ok 2024-01-16T04:52:02.089 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_zero ... ok 2024-01-16T04:52:02.145 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_none ... ok 2024-01-16T04:52:02.202 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_none ... ok 2024-01-16T04:52:02.428 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_account_usage ... ok 2024-01-16T04:52:02.462 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_head_bucket_usage ... ok 2024-01-16T04:52:02.873 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_unordered ... ok 2024-01-16T04:52:03.252 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_unordered ... ok 2024-01-16T04:52:03.308 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_invalid ... ok 2024-01-16T04:52:03.363 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_none ... ok 2024-01-16T04:52:03.418 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_empty ... ok 2024-01-16T04:52:03.473 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken_empty ... ok 2024-01-16T04:52:03.572 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken ... ok 2024-01-16T04:52:03.672 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_both_continuationtoken_startafter ... ok 2024-01-16T04:52:03.727 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_unreadable ... ok 2024-01-16T04:52:03.781 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_unreadable ... ok 2024-01-16T04:52:03.837 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_not_in_list ... ok 2024-01-16T04:52:03.891 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_not_in_list ... ok 2024-01-16T04:52:03.973 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_after_list ... ok 2024-01-16T04:52:04.027 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_after_list ... ok 2024-01-16T04:52:04.258 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data ... ok 2024-01-16T04:52:04.565 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data_versioning ... ok 2024-01-16T04:52:04.599 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous ... ok 2024-01-16T04:52:04.632 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous ... ok 2024-01-16T04:52:04.656 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous_fail ... ok 2024-01-16T04:52:04.680 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous_fail ... ok 2024-01-16T04:52:04.689 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_notexist ... ok 2024-01-16T04:52:04.699 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_notexist ... ok 2024-01-16T04:52:04.708 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_notexist ... ok 2024-01-16T04:52:04.748 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_nonempty ... ok 2024-01-16T04:52:04.899 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_concurrent_set_canned_acl ... ok 2024-01-16T04:52:04.909 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_write_to_nonexist_bucket ... ok 2024-01-16T04:52:05.001 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_delete ... ok 2024-01-16T04:52:05.028 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_read_not_exist ... ok 2024-01-16T04:52:05.054 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_requestid_matches_header_on_error ... ok 2024-01-16T04:52:05.277 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete ... ok 2024-01-16T04:52:05.501 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete ... ok 2024-01-16T04:52:05.532 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_head_zero_bytes ... ok 2024-01-16T04:52:05.560 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_write_check_etag ... ok 2024-01-16T04:52:05.592 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_write_cache_control ... ok 2024-01-16T04:52:05.624 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_write_expires ... ok 2024-01-16T04:52:05.752 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_write_read_update_read_delete ... ok 2024-01-16T04:52:05.784 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_good ... ok 2024-01-16T04:52:05.817 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_empty ... ok 2024-01-16T04:52:05.890 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_overwrite_to_empty ... ok 2024-01-16T04:52:05.929 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_metadata_replaced_on_put ... ok 2024-01-16T04:52:06.000 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_write_file ... ok 2024-01-16T04:52:06.066 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_anonymous_request ... ok 2024-01-16T04:52:06.099 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request ... ok 2024-01-16T04:52:06.166 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_no_content_type ... ok 2024-01-16T04:52:06.186 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request_bad_access_key ... ok 2024-01-16T04:52:06.208 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_success_code ... ok 2024-01-16T04:52:06.230 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_invalid_success_code ... ok 2024-01-16T04:52:06.301 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_larger_than_chunk ... ok 2024-01-16T04:52:06.332 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_key_from_filename ... ok 2024-01-16T04:52:06.357 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_ignored_header ... ok 2024-01-16T04:52:06.384 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_case_insensitive_condition_fields ... ok 2024-01-16T04:52:06.415 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_escaped_field_values ... ok 2024-01-16T04:52:06.485 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_success_redirect_action ... ok 2024-01-16T04:52:06.508 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_signature ... ok 2024-01-16T04:52:06.531 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_access_key ... ok 2024-01-16T04:52:06.553 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_date_format ... ok 2024-01-16T04:52:06.576 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_no_key_specified ... ok 2024-01-16T04:52:06.598 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_signature ... ok 2024-01-16T04:52:06.621 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_policy_condition ... ok 2024-01-16T04:52:06.652 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_user_specified_header ... ok 2024-01-16T04:52:06.675 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_request_missing_policy_specified_field ... ok 2024-01-16T04:52:06.698 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_condition_is_case_sensitive ... ok 2024-01-16T04:52:06.721 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_expires_is_case_sensitive ... ok 2024-01-16T04:52:06.744 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_expired_policy ... ok 2024-01-16T04:52:06.767 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_request_field_value ... ok 2024-01-16T04:52:06.790 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_expires_condition ... ok 2024-01-16T04:52:06.812 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_conditions_list ... ok 2024-01-16T04:52:06.835 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_limit_exceeded ... ok 2024-01-16T04:52:06.859 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_content_length_argument ... ok 2024-01-16T04:52:06.881 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_content_length_argument ... ok 2024-01-16T04:52:06.904 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_below_minimum ... ok 2024-01-16T04:52:06.928 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_empty_conditions ... ok 2024-01-16T04:52:06.999 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_good ... ok 2024-01-16T04:52:07.072 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_failed ... ok 2024-01-16T04:52:07.106 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_good ... ok 2024-01-16T04:52:07.178 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_failed ... ok 2024-01-16T04:52:07.251 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_good ... ok 2024-01-16T04:52:08.291 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_failed ... ok 2024-01-16T04:52:08.404 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_good ... ok 2024-01-16T04:52:08.478 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_failed ... ok 2024-01-16T04:52:08.600 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_good ... ok 2024-01-16T04:52:08.762 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_failed ... ok 2024-01-16T04:52:08.883 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_overwrite_existed_good ... ok 2024-01-16T04:52:08.994 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_nonexisted_failed ... ok 2024-01-16T04:52:09.115 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_good ... ok 2024-01-16T04:52:09.276 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_failed ... ok 2024-01-16T04:52:09.349 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_nonexisted_good ... ok 2024-01-16T04:52:09.510 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_overwrite_existed_failed ... ok 2024-01-16T04:52:09.543 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get ... ok 2024-01-16T04:52:09.614 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_gone ... ok 2024-01-16T04:52:09.682 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_delete_key_bucket_gone ... ok 2024-01-16T04:52:09.725 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_gone ... ok 2024-01-16T04:52:09.749 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_head ... ok 2024-01-16T04:52:09.758 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_notexist ... ok 2024-01-16T04:52:09.819 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_extended ... ok 2024-01-16T04:52:09.850 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_acl ... ok 2024-01-16T04:52:09.882 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_acl ... ok 2024-01-16T04:52:09.914 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated ... ok 2024-01-16T04:52:09.947 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_response_headers ... ok 2024-01-16T04:52:09.980 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_acl ... ok 2024-01-16T04:52:10.011 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_acl ... ok 2024-01-16T04:52:10.117 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_gone ... ok 2024-01-16T04:52:10.228 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_gone ... ok 2024-01-16T04:52:10.258 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired ... ok 2024-01-16T04:52:10.287 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired_tenant ... ok 2024-01-16T04:52:10.318 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_range_zero ... ok 2024-01-16T04:52:10.349 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_max_range ... ok 2024-01-16T04:52:10.379 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_positive_range ... ok 2024-01-16T04:52:10.416 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put ... ok 2024-01-16T04:52:10.456 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put_write_access ... ok 2024-01-16T04:52:10.483 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_put_authenticated ... ok 2024-01-16T04:52:10.514 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_raw_put_authenticated_expired ... ok 2024-01-16T04:52:10.524 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_one ... ok 2024-01-16T04:52:10.535 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_two ... ok 2024-01-16T04:52:10.545 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_ip ... ok 2024-01-16T04:52:10.554 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_dns_underscore ... ok 2024-01-16T04:52:10.579 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists ... ok 2024-01-16T04:52:10.639 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_get_location ... ok 2024-01-16T04:52:10.664 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists_nonowner ... ok 2024-01-16T04:52:10.687 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_overwrite_acl ... FAIL 2024-01-16T04:52:10.711 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_new_acl ... FAIL 2024-01-16T04:52:10.736 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_default ... ok 2024-01-16T04:52:10.796 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_during_create ... ok 2024-01-16T04:52:10.906 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned ... ok 2024-01-16T04:52:10.968 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_publicreadwrite ... ok 2024-01-16T04:52:11.029 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_authenticatedread ... ok 2024-01-16T04:52:11.101 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_acl_default ... ok 2024-01-16T04:52:11.173 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_during_create ... ok 2024-01-16T04:52:11.296 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned ... ok 2024-01-16T04:52:11.368 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_publicreadwrite ... ok 2024-01-16T04:52:11.440 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_authenticatedread ... ok 2024-01-16T04:52:11.564 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerread ... ok 2024-01-16T04:52:11.685 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerfullcontrol ... ok 2024-01-16T04:52:11.770 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_owner ... ok 2024-01-16T04:52:11.819 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_attributes ... ok 2024-01-16T04:52:11.871 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_private_to_private ... ok 2024-01-16T04:52:11.993 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_acl ... ok 2024-01-16T04:52:12.114 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_acl_write ... ok 2024-01-16T04:52:12.240 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_acl_writeacp ... ok 2024-01-16T04:52:12.362 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_acl_read ... ok 2024-01-16T04:52:12.485 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_acl_readacp ... ok 2024-01-16T04:52:12.604 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_fullcontrol ... ok 2024-01-16T04:52:12.714 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_read ... ok 2024-01-16T04:52:12.825 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_readacp ... ok 2024-01-16T04:52:12.939 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_write ... ok 2024-01-16T04:52:13.052 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_writeacp ... ok 2024-01-16T04:52:13.086 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_nonexist_user ... ok 2024-01-16T04:52:13.189 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_no_grants ... ok 2024-01-16T04:52:13.264 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_header_acl_grants ... ok 2024-01-16T04:52:13.341 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_header_acl_grants ... ok 2024-01-16T04:52:13.420 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email ... ok 2024-01-16T04:52:13.453 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email_not_exist ... ok 2024-01-16T04:52:13.577 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_revoke_all ... ok 2024-01-16T04:52:13.782 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_private ... ok 2024-01-16T04:52:14.012 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_private ... ok 2024-01-16T04:52:14.221 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicread ... ok 2024-01-16T04:52:14.429 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicread ... ok 2024-01-16T04:52:14.633 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicreadwrite ... ok 2024-01-16T04:52:14.838 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicreadwrite ... ok 2024-01-16T04:52:15.046 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_private ... ok 2024-01-16T04:52:15.253 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicread ... ok 2024-01-16T04:52:15.460 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicreadwrite ... ok 2024-01-16T04:52:15.625 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_private ... ok 2024-01-16T04:52:15.824 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicread ... ok 2024-01-16T04:52:15.989 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicreadwrite ... ok 2024-01-16T04:52:16.157 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_buckets_create_then_list ... ok 2024-01-16T04:52:16.164 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_anonymous ... ok 2024-01-16T04:52:16.173 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_invalid_auth ... ok 2024-01-16T04:52:16.180 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_bad_auth ... ok 2024-01-16T04:52:16.428 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_alpha ... ok 2024-01-16T04:52:16.678 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_digit ... ok 2024-01-16T04:52:16.695 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_period ... ok 2024-01-16T04:52:16.712 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_hyphen ... ok 2024-01-16T04:52:16.776 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_not_overriding ... ok 2024-01-16T04:52:17.438 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_special_key_names ... ok 2024-01-16T04:52:17.511 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_special_prefix ... ok 2024-01-16T04:52:17.608 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_copy_zero_size ... ok 2024-01-16T04:52:17.735 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_copy_same_bucket ... ok 2024-01-16T04:52:17.863 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_copy_verify_contenttype ... ok 2024-01-16T04:52:17.944 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself ... ok 2024-01-16T04:52:18.025 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself_with_metadata ... ok 2024-01-16T04:52:18.197 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_copy_diff_bucket ... ok 2024-01-16T04:52:18.242 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_bucket ... ok 2024-01-16T04:52:18.355 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_object_bucket ... ok 2024-01-16T04:52:18.492 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_copy_canned_acl ... ok 2024-01-16T04:52:18.714 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_copy_retaining_metadata ... ok 2024-01-16T04:52:18.896 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_copy_replacing_metadata ... ok 2024-01-16T04:52:18.924 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_copy_bucket_not_found ... ok 2024-01-16T04:52:18.953 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_copy_key_not_found ... ok 2024-01-16T04:52:19.520 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_bucket ... ok 2024-01-16T04:52:19.665 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_url_encoding ... ok 2024-01-16T04:52:20.692 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioning_multipart_upload ... ok 2024-01-16T04:52:20.730 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_empty ... ok 2024-01-16T04:52:20.787 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_small ... ok 2024-01-16T04:52:21.052 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_small ... ok 2024-01-16T04:52:21.168 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_invalid_range ... ok 2024-01-16T04:52:21.386 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_without_range ... ok 2024-01-16T04:52:22.319 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_special_names ... ok 2024-01-16T04:52:23.004 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload ... ok 2024-01-16T04:52:24.463 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_versioned ... ok 2024-01-16T04:52:28.534 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_resend_part ... ok 2024-01-16T04:52:29.311 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_multiple_sizes ... ok 2024-01-16T04:52:30.663 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_multiple_sizes ... ok 2024-01-16T04:52:30.792 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_size_too_small ... ok 2024-01-16T04:52:31.119 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_contents ... ok 2024-01-16T04:52:31.439 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_overwrite_existing_object ... ok 2024-01-16T04:52:31.606 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload ... ok 2024-01-16T04:52:31.680 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload_not_found ... ok 2024-01-16T04:52:31.992 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload ... ok 2024-01-16T04:52:32.077 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_missing_part ... ok 2024-01-16T04:52:32.164 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_incorrect_etag ... ok 2024-01-16T04:52:32.188 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_100_continue ... ok 2024-01-16T04:52:32.311 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_set_cors ... ok 2024-01-16T04:52:35.427 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_response ... ok 2024-01-16T04:52:38.470 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_wildcard ... ok 2024-01-16T04:52:41.515 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_cors_header_option ... ok 2024-01-16T04:52:41.562 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_cors_presigned_get_object ... ok 2024-01-16T04:52:41.598 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_cors_presigned_get_object_tenant ... ok 2024-01-16T04:52:41.722 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_set_bucket_tagging ... ok 2024-01-16T04:52:41.816 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_1mb ... ok 2024-01-16T04:52:41.983 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_4mb ... ok 2024-01-16T04:52:42.246 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_8mb ... ok 2024-01-16T04:52:42.385 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_1mb ... ok 2024-01-16T04:52:42.608 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_4mb ... ok 2024-01-16T04:52:43.000 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_8mb ... ok 2024-01-16T04:52:43.113 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_1mb ... ok 2024-01-16T04:52:43.358 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_4mb ... ok 2024-01-16T04:52:43.773 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_8mb ... ok 2024-01-16T04:52:43.893 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_atomic_conditional_write_1mb ... ok 2024-01-16T04:52:44.063 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_atomic_multipart_upload_write ... ok 2024-01-16T04:52:44.324 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_multipart_resend_first_finishes_last ... ok 2024-01-16T04:52:44.398 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_response_code ... ok 2024-01-16T04:52:50.024 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_ranged_big_request_response_code ... ok 2024-01-16T04:52:50.099 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_skip_leading_bytes_response_code ... ok 2024-01-16T04:52:50.171 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_return_trailing_bytes_response_code ... ok 2024-01-16T04:52:50.246 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_invalid_range ... ok 2024-01-16T04:52:50.323 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_empty_object ... ok 2024-01-16T04:52:50.557 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_create_suspend ... ok 2024-01-16T04:52:58.088 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove ... ok 2024-01-16T04:52:58.638 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove_head ... ok 2024-01-16T04:52:58.857 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_removal ... ok 2024-01-16T04:52:59.154 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite ... ok 2024-01-16T04:52:59.493 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite_suspended ... ok 2024-01-16T04:53:02.436 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_suspend_versions ... ok 2024-01-16T04:53:06.028 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_all ... ok 2024-01-16T04:53:20.191 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_special_names ... ok 2024-01-16T04:53:21.578 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_overwrite_multipart ... ok 2024-01-16T04:53:22.230 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_list_marker ... ok 2024-01-16T04:53:23.180 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_copy_obj_version ... ok 2024-01-16T04:53:23.571 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete ... ok 2024-01-16T04:53:23.976 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker ... ok 2024-01-16T04:53:24.105 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker_create ... ok 2024-01-16T04:53:24.536 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl ... ok 2024-01-16T04:53:24.875 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl_no_version_specified ... ok 2024-01-16T04:53:25.183 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioned_concurrent_object_create_and_remove ... ok 2024-01-16T04:53:25.213 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set ... ok 2024-01-16T04:53:25.287 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get ... ok 2024-01-16T04:53:25.361 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get_no_id ... ok 2024-01-16T04:53:25.388 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_id_too_long ... ok 2024-01-16T04:53:25.415 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_same_id ... ok 2024-01-16T04:53:25.529 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_invalid_status ... ok 2024-01-16T04:53:25.559 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_date ... ok 2024-01-16T04:53:25.587 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_invalid_date ... ok 2024-01-16T04:53:25.633 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent ... ok 2024-01-16T04:53:25.664 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_deletemarker ... ok 2024-01-16T04:53:25.694 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_filter ... ok 2024-01-16T04:53:25.724 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_empty_filter ... ok 2024-01-16T04:53:25.755 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_multipart ... ok 2024-01-16T04:53:25.830 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1b ... ok 2024-01-16T04:53:25.906 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1kb ... ok 2024-01-16T04:53:25.956 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1MB ... ok 2024-01-16T04:53:26.031 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_13b ... ok 2024-01-16T04:53:26.075 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_method_head ... ok 2024-01-16T04:53:26.150 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_present ... ok 2024-01-16T04:53:26.234 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_other_key ... ok 2024-01-16T04:53:26.301 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_invalid_md5 ... ok 2024-01-16T04:53:26.370 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_md5 ... ok 2024-01-16T04:53:26.437 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_key ... ok 2024-01-16T04:53:26.507 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_encryption_key_no_sse_c ... ok 2024-01-16T04:53:27.341 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_upload ... ok 2024-01-16T04:53:27.872 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_bad_download ... ok 2024-01-16T04:53:27.936 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_post_object_authenticated_request ... ok 2024-01-16T04:53:27.981 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_method_head ... ok 2024-01-16T04:53:28.057 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_present ... ok 2024-01-16T04:53:28.124 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_no_key ... ok 2024-01-16T04:53:28.191 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_not_declared ... ok 2024-01-16T04:53:29.031 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_upload ... ok 2024-01-16T04:53:29.492 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_1 ... ok 2024-01-16T04:53:29.948 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_2 ... ok 2024-01-16T04:53:29.984 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_post_object_authenticated_request ... ok 2024-01-16T04:53:30.060 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1b ... ok 2024-01-16T04:53:30.134 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1kb ... ok 2024-01-16T04:53:30.186 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1MB ... ok 2024-01-16T04:53:30.261 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_13b ... ok 2024-01-16T04:53:30.337 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_read_declare ... ok 2024-01-16T04:53:30.380 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy ... ok 2024-01-16T04:53:30.425 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy ... ok 2024-01-16T04:53:30.483 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_acl ... ok 2024-01-16T04:53:30.542 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_acl ... ok 2024-01-16T04:53:30.667 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_another_bucket ... ok 2024-01-16T04:53:30.790 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_another_bucket ... ok 2024-01-16T04:53:30.955 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_obj_tagging ... ok 2024-01-16T04:53:31.072 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_obj_head_tagging ... ok 2024-01-16T04:53:31.257 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_max_tags ... ok 2024-01-16T04:53:31.413 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_excess_tags ... ok 2024-01-16T04:53:31.573 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_max_kvsize_tags ... ok 2024-01-16T04:53:31.732 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_excess_key_tags ... ok 2024-01-16T04:53:31.889 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_excess_val_tags ... ok 2024-01-16T04:53:32.100 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_modify_tags ... ok 2024-01-16T04:53:32.311 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_delete_tags ... ok 2024-01-16T04:53:32.422 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_anonymous_request ... ok 2024-01-16T04:53:32.457 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_authenticated_request ... ok 2024-01-16T04:53:32.574 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_obj_with_tags ... ok 2024-01-16T04:53:32.704 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_tags_acl_public ... ok 2024-01-16T04:53:32.870 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_tags_acl_public ... ok 2024-01-16T04:53:33.047 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_delete_tags_obj_public ... ok 2024-01-16T04:53:33.283 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_atomic_upload_return_version_id ... ok 2024-01-16T04:53:34.819 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_multipart_upload_return_version_id ... ok 2024-01-16T04:53:34.944 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_existing_tag ... ok 2024-01-16T04:53:35.178 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_tagging_existing_tag ... ok 2024-01-16T04:53:35.347 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_tagging_existing_tag ... ok 2024-01-16T04:53:35.613 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source ... ok 2024-01-16T04:53:35.781 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source_meta ... ok 2024-01-16T04:53:35.866 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_acl ... ok 2024-01-16T04:53:36.025 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_grant ... ok 2024-01-16T04:53:36.235 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_acl_existing_tag ... ok 2024-01-16T04:53:36.308 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock ... ok 2024-01-16T04:53:36.373 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_bucket ... ok 2024-01-16T04:53:36.436 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_with_days_and_years ... ok 2024-01-16T04:53:36.501 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_days ... ok 2024-01-16T04:53:36.610 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_years ... ok 2024-01-16T04:53:36.674 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_status ... ok 2024-01-16T04:53:36.737 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_suspend_versioning ... ok 2024-01-16T04:53:36.804 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock ... ok 2024-01-16T04:53:36.870 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock_invalid_bucket ... ok 2024-01-16T04:53:36.919 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention ... ok 2024-01-16T04:53:36.989 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_bucket ... ok 2024-01-16T04:53:37.109 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_mode ... ok 2024-01-16T04:53:37.205 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention ... ok 2024-01-16T04:53:37.276 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_invalid_bucket ... ok 2024-01-16T04:53:37.380 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_versionid ... ok 2024-01-16T04:53:37.477 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_override_default_retention ... ok 2024-01-16T04:53:37.579 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_increase_period ... ok 2024-01-16T04:53:37.675 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period ... ok 2024-01-16T04:53:37.775 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period_bypass ... ok 2024-01-16T04:53:37.870 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention ... ok 2024-01-16T04:53:37.914 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold ... ok 2024-01-16T04:53:38.014 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_bucket ... ok 2024-01-16T04:53:38.091 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_status ... ok 2024-01-16T04:53:38.222 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold ... ok 2024-01-16T04:53:38.295 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold_invalid_bucket ... ok 2024-01-16T04:53:38.383 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_on ... ok 2024-01-16T04:53:38.432 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_off ... ok 2024-01-16T04:53:38.497 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_metadata ... ok 2024-01-16T04:53:38.552 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_object_lock_uploading_obj ... ok 2024-01-16T04:53:38.673 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifmatch_good ... ok 2024-01-16T04:53:38.795 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifnonematch_failed ... ok 2024-01-16T04:53:38.814 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_user_policy ... ok 2024-01-16T04:53:38.840 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_policy_status ... ok 2024-01-16T04:53:38.915 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_public_acl_bucket_policy_status ... ok 2024-01-16T04:53:38.990 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_authpublic_acl_bucket_policy_status ... ok 2024-01-16T04:53:39.066 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_publicpolicy_acl_bucket_policy_status ... ok 2024-01-16T04:53:39.144 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_acl_bucket_policy_status ... ok 2024-01-16T04:53:39.216 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_deny_bucket_policy_status ... ok 2024-01-16T04:53:39.243 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_get_default_public_block ... ok 2024-01-16T04:53:39.316 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_put_public_block ... ok 2024-01-16T04:53:39.520 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_block_public_put_bucket_acls ... ok 2024-01-16T04:53:39.680 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_block_public_object_canned_acls ... ok 2024-01-16T04:53:39.754 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_block_public_policy ... ok 2024-01-16T04:53:39.941 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_s3.test_ignore_public_acls ... ok 2024-01-16T04:55:05.366 INFO:teuthology.orchestra.run.smithi156.stderr:s3tests_boto3.functional.test_utils.test_generate ... ok 2024-01-16T04:55:05.366 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:55:05.366 INFO:teuthology.orchestra.run.smithi156.stderr:====================================================================== 2024-01-16T04:55:05.366 INFO:teuthology.orchestra.run.smithi156.stderr:FAIL: s3tests_boto3.functional.test_s3.test_bucket_recreate_overwrite_acl 2024-01-16T04:55:05.366 INFO:teuthology.orchestra.run.smithi156.stderr:---------------------------------------------------------------------- 2024-01-16T04:55:05.366 INFO:teuthology.orchestra.run.smithi156.stderr:Traceback (most recent call last): 2024-01-16T04:55:05.366 INFO:teuthology.orchestra.run.smithi156.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2024-01-16T04:55:05.366 INFO:teuthology.orchestra.run.smithi156.stderr: self.test(*self.arg) 2024-01-16T04:55:05.366 INFO:teuthology.orchestra.run.smithi156.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 4439, in test_bucket_recreate_overwrite_acl 2024-01-16T04:55:05.367 INFO:teuthology.orchestra.run.smithi156.stderr: e = assert_raises(ClientError, client.create_bucket, Bucket=bucket_name) 2024-01-16T04:55:05.367 INFO:teuthology.orchestra.run.smithi156.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/utils.py", line 21, in assert_raises 2024-01-16T04:55:05.367 INFO:teuthology.orchestra.run.smithi156.stderr: raise AssertionError("%s not raised" % excName) 2024-01-16T04:55:05.367 INFO:teuthology.orchestra.run.smithi156.stderr:AssertionError: ClientError not raised 2024-01-16T04:55:05.367 INFO:teuthology.orchestra.run.smithi156.stderr:-------------------- >> begin captured logging << -------------------- 2024-01-16T04:55:05.367 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2024-01-16T04:55:05.367 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-01-16T04:55:05.367 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7f8a24577598> 2024-01-16T04:55:05.367 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-01-16T04:55:05.367 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60) 2024-01-16T04:55:05.367 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.client: DEBUG: Registering retry handlers for service: s3 2024-01-16T04:55:05.367 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:55:05.367 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.367 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.368 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:55:05.368 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:55:05.368 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.368 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:55:05.368 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:55:05.368 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-kjdceoogenvalx4-211', 'query_string': {}, 'method': 'PUT', 'headers': {'x-amz-acl': 'public-read', 'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-532.el8.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi156.front.sepia.ceph.com:80/test-client.0-kjdceoogenvalx4-211', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-kjdceoogenvalx4-211'}}} 2024-01-16T04:55:05.368 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.368 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.368 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-01-16T04:55:05.368 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.368 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.utils: DEBUG: Using S3 path style addressing. 2024-01-16T04:55:05.368 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-01-16T04:55:05.368 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-01-16T04:55:05.369 INFO:teuthology.orchestra.run.smithi156.stderr:PUT 2024-01-16T04:55:05.369 INFO:teuthology.orchestra.run.smithi156.stderr:/test-client.0-kjdceoogenvalx4-211 2024-01-16T04:55:05.369 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:55:05.369 INFO:teuthology.orchestra.run.smithi156.stderr:host:smithi156.front.sepia.ceph.com 2024-01-16T04:55:05.369 INFO:teuthology.orchestra.run.smithi156.stderr:x-amz-acl:public-read 2024-01-16T04:55:05.369 INFO:teuthology.orchestra.run.smithi156.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:55:05.369 INFO:teuthology.orchestra.run.smithi156.stderr:x-amz-date:20240116T045210Z 2024-01-16T04:55:05.369 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:55:05.369 INFO:teuthology.orchestra.run.smithi156.stderr:host;x-amz-acl;x-amz-content-sha256;x-amz-date 2024-01-16T04:55:05.369 INFO:teuthology.orchestra.run.smithi156.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:55:05.369 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.auth: DEBUG: StringToSign: 2024-01-16T04:55:05.369 INFO:teuthology.orchestra.run.smithi156.stderr:AWS4-HMAC-SHA256 2024-01-16T04:55:05.369 INFO:teuthology.orchestra.run.smithi156.stderr:20240116T045210Z 2024-01-16T04:55:05.369 INFO:teuthology.orchestra.run.smithi156.stderr:20240116/us-east-1/s3/aws4_request 2024-01-16T04:55:05.370 INFO:teuthology.orchestra.run.smithi156.stderr:9e433e2f0c7390241ea4910aa0b0de5950a88414f363b5dfc85cbd43027d7939 2024-01-16T04:55:05.370 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.auth: DEBUG: Signature: 2024-01-16T04:55:05.370 INFO:teuthology.orchestra.run.smithi156.stderr:ed4ce9f9065caf086466a298db5b7b423f0a0c1b98d670e1cf4ecbeb5cbb3173 2024-01-16T04:55:05.370 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-01-16T04:55:05.370 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-01-16T04:55:05.370 INFO:teuthology.orchestra.run.smithi156.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi156.front.sepia.ceph.com:80 2024-01-16T04:55:05.370 INFO:teuthology.orchestra.run.smithi156.stderr:urllib3.connectionpool: DEBUG: http://smithi156.front.sepia.ceph.com:80 "PUT /test-client.0-kjdceoogenvalx4-211 HTTP/1.1" 200 0 2024-01-16T04:55:05.370 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000e3526d9fe3f009e9-0065a60b7a-1061-default', 'Content-Length': '0', 'Date': 'Tue, 16 Jan 2024 04:52:10 GMT'} 2024-01-16T04:55:05.370 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.parsers: DEBUG: Response body: 2024-01-16T04:55:05.370 INFO:teuthology.orchestra.run.smithi156.stderr:b'' 2024-01-16T04:55:05.370 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-01-16T04:55:05.370 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-01-16T04:55:05.370 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.370 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:55:05.370 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.371 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.371 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:55:05.371 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:55:05.371 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.371 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:55:05.371 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:55:05.371 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-kjdceoogenvalx4-211', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-532.el8.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi156.front.sepia.ceph.com:80/test-client.0-kjdceoogenvalx4-211', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-kjdceoogenvalx4-211'}}} 2024-01-16T04:55:05.371 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.371 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.371 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-01-16T04:55:05.371 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.371 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-01-16T04:55:05.371 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-01-16T04:55:05.372 INFO:teuthology.orchestra.run.smithi156.stderr:PUT 2024-01-16T04:55:05.372 INFO:teuthology.orchestra.run.smithi156.stderr:/test-client.0-kjdceoogenvalx4-211 2024-01-16T04:55:05.372 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:55:05.372 INFO:teuthology.orchestra.run.smithi156.stderr:host:smithi156.front.sepia.ceph.com 2024-01-16T04:55:05.372 INFO:teuthology.orchestra.run.smithi156.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:55:05.372 INFO:teuthology.orchestra.run.smithi156.stderr:x-amz-date:20240116T045210Z 2024-01-16T04:55:05.372 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:55:05.372 INFO:teuthology.orchestra.run.smithi156.stderr:host;x-amz-content-sha256;x-amz-date 2024-01-16T04:55:05.372 INFO:teuthology.orchestra.run.smithi156.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:55:05.372 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.auth: DEBUG: StringToSign: 2024-01-16T04:55:05.372 INFO:teuthology.orchestra.run.smithi156.stderr:AWS4-HMAC-SHA256 2024-01-16T04:55:05.372 INFO:teuthology.orchestra.run.smithi156.stderr:20240116T045210Z 2024-01-16T04:55:05.372 INFO:teuthology.orchestra.run.smithi156.stderr:20240116/us-east-1/s3/aws4_request 2024-01-16T04:55:05.372 INFO:teuthology.orchestra.run.smithi156.stderr:d4d2702f2a6811419acaa9693965cc23842c39b7b001810bd8ceadedae4f744f 2024-01-16T04:55:05.372 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.auth: DEBUG: Signature: 2024-01-16T04:55:05.373 INFO:teuthology.orchestra.run.smithi156.stderr:093740f4af90dcc9d4c2576e6d3c4d9eeb261dbb7b6934fcfdb310acd0b300f2 2024-01-16T04:55:05.373 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-01-16T04:55:05.373 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-01-16T04:55:05.373 INFO:teuthology.orchestra.run.smithi156.stderr:urllib3.connectionpool: DEBUG: http://smithi156.front.sepia.ceph.com:80 "PUT /test-client.0-kjdceoogenvalx4-211 HTTP/1.1" 200 0 2024-01-16T04:55:05.373 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx000003f40d1e97427ee2c-0065a60b7a-1061-default', 'Content-Length': '0', 'Date': 'Tue, 16 Jan 2024 04:52:10 GMT'} 2024-01-16T04:55:05.373 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.parsers: DEBUG: Response body: 2024-01-16T04:55:05.373 INFO:teuthology.orchestra.run.smithi156.stderr:b'' 2024-01-16T04:55:05.373 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-01-16T04:55:05.373 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-01-16T04:55:05.373 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.373 INFO:teuthology.orchestra.run.smithi156.stderr:--------------------- >> end captured logging << --------------------- 2024-01-16T04:55:05.373 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:55:05.373 INFO:teuthology.orchestra.run.smithi156.stderr:====================================================================== 2024-01-16T04:55:05.373 INFO:teuthology.orchestra.run.smithi156.stderr:FAIL: s3tests_boto3.functional.test_s3.test_bucket_recreate_new_acl 2024-01-16T04:55:05.374 INFO:teuthology.orchestra.run.smithi156.stderr:---------------------------------------------------------------------- 2024-01-16T04:55:05.374 INFO:teuthology.orchestra.run.smithi156.stderr:Traceback (most recent call last): 2024-01-16T04:55:05.374 INFO:teuthology.orchestra.run.smithi156.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2024-01-16T04:55:05.374 INFO:teuthology.orchestra.run.smithi156.stderr: self.test(*self.arg) 2024-01-16T04:55:05.374 INFO:teuthology.orchestra.run.smithi156.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 4453, in test_bucket_recreate_new_acl 2024-01-16T04:55:05.374 INFO:teuthology.orchestra.run.smithi156.stderr: e = assert_raises(ClientError, client.create_bucket, Bucket=bucket_name, ACL='public-read') 2024-01-16T04:55:05.374 INFO:teuthology.orchestra.run.smithi156.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/utils.py", line 21, in assert_raises 2024-01-16T04:55:05.374 INFO:teuthology.orchestra.run.smithi156.stderr: raise AssertionError("%s not raised" % excName) 2024-01-16T04:55:05.374 INFO:teuthology.orchestra.run.smithi156.stderr:AssertionError: ClientError not raised 2024-01-16T04:55:05.374 INFO:teuthology.orchestra.run.smithi156.stderr:-------------------- >> begin captured logging << -------------------- 2024-01-16T04:55:05.374 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2024-01-16T04:55:05.374 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-01-16T04:55:05.374 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7f8a24577598> 2024-01-16T04:55:05.375 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2024-01-16T04:55:05.375 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60) 2024-01-16T04:55:05.375 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.client: DEBUG: Registering retry handlers for service: s3 2024-01-16T04:55:05.375 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:55:05.375 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.375 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.375 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:55:05.375 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:55:05.375 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.375 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:55:05.375 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:55:05.375 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-kjdceoogenvalx4-212', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-532.el8.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi156.front.sepia.ceph.com:80/test-client.0-kjdceoogenvalx4-212', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-kjdceoogenvalx4-212'}}} 2024-01-16T04:55:05.375 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.375 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.376 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-01-16T04:55:05.376 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.376 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.utils: DEBUG: Using S3 path style addressing. 2024-01-16T04:55:05.376 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-01-16T04:55:05.376 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-01-16T04:55:05.376 INFO:teuthology.orchestra.run.smithi156.stderr:PUT 2024-01-16T04:55:05.376 INFO:teuthology.orchestra.run.smithi156.stderr:/test-client.0-kjdceoogenvalx4-212 2024-01-16T04:55:05.376 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:55:05.376 INFO:teuthology.orchestra.run.smithi156.stderr:host:smithi156.front.sepia.ceph.com 2024-01-16T04:55:05.376 INFO:teuthology.orchestra.run.smithi156.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:55:05.376 INFO:teuthology.orchestra.run.smithi156.stderr:x-amz-date:20240116T045210Z 2024-01-16T04:55:05.376 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:55:05.377 INFO:teuthology.orchestra.run.smithi156.stderr:host;x-amz-content-sha256;x-amz-date 2024-01-16T04:55:05.377 INFO:teuthology.orchestra.run.smithi156.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:55:05.377 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.auth: DEBUG: StringToSign: 2024-01-16T04:55:05.377 INFO:teuthology.orchestra.run.smithi156.stderr:AWS4-HMAC-SHA256 2024-01-16T04:55:05.377 INFO:teuthology.orchestra.run.smithi156.stderr:20240116T045210Z 2024-01-16T04:55:05.377 INFO:teuthology.orchestra.run.smithi156.stderr:20240116/us-east-1/s3/aws4_request 2024-01-16T04:55:05.377 INFO:teuthology.orchestra.run.smithi156.stderr:05ab617af70fb5379449c93ede770981d7c9802623b9a8b777d161ff897dcb09 2024-01-16T04:55:05.377 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.auth: DEBUG: Signature: 2024-01-16T04:55:05.377 INFO:teuthology.orchestra.run.smithi156.stderr:1bd6d2c7c7a5d3144346299612376a55a26463403f4bc61b85f745aac8e054aa 2024-01-16T04:55:05.377 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-01-16T04:55:05.377 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-01-16T04:55:05.377 INFO:teuthology.orchestra.run.smithi156.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi156.front.sepia.ceph.com:80 2024-01-16T04:55:05.377 INFO:teuthology.orchestra.run.smithi156.stderr:urllib3.connectionpool: DEBUG: http://smithi156.front.sepia.ceph.com:80 "PUT /test-client.0-kjdceoogenvalx4-212 HTTP/1.1" 200 0 2024-01-16T04:55:05.377 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000757147036a171238-0065a60b7a-1061-default', 'Content-Length': '0', 'Date': 'Tue, 16 Jan 2024 04:52:10 GMT'} 2024-01-16T04:55:05.378 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.parsers: DEBUG: Response body: 2024-01-16T04:55:05.378 INFO:teuthology.orchestra.run.smithi156.stderr:b'' 2024-01-16T04:55:05.378 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-01-16T04:55:05.378 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-01-16T04:55:05.378 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.378 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:55:05.378 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.378 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.378 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2024-01-16T04:55:05.378 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:55:05.378 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.378 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:55:05.378 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2024-01-16T04:55:05.378 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-kjdceoogenvalx4-212', 'query_string': {}, 'method': 'PUT', 'headers': {'x-amz-acl': 'public-read', 'User-Agent': 'Boto3/1.23.10 Python/3.6.8 Linux/4.18.0-532.el8.x86_64 Botocore/1.26.10'}, 'body': b'', 'url': 'http://smithi156.front.sepia.ceph.com:80/test-client.0-kjdceoogenvalx4-212', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-kjdceoogenvalx4-212'}}} 2024-01-16T04:55:05.379 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.379 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.379 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2024-01-16T04:55:05.379 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.379 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2024-01-16T04:55:05.379 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.auth: DEBUG: CanonicalRequest: 2024-01-16T04:55:05.379 INFO:teuthology.orchestra.run.smithi156.stderr:PUT 2024-01-16T04:55:05.379 INFO:teuthology.orchestra.run.smithi156.stderr:/test-client.0-kjdceoogenvalx4-212 2024-01-16T04:55:05.379 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:55:05.379 INFO:teuthology.orchestra.run.smithi156.stderr:host:smithi156.front.sepia.ceph.com 2024-01-16T04:55:05.380 INFO:teuthology.orchestra.run.smithi156.stderr:x-amz-acl:public-read 2024-01-16T04:55:05.380 INFO:teuthology.orchestra.run.smithi156.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:55:05.380 INFO:teuthology.orchestra.run.smithi156.stderr:x-amz-date:20240116T045210Z 2024-01-16T04:55:05.380 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:55:05.380 INFO:teuthology.orchestra.run.smithi156.stderr:host;x-amz-acl;x-amz-content-sha256;x-amz-date 2024-01-16T04:55:05.380 INFO:teuthology.orchestra.run.smithi156.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-01-16T04:55:05.380 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.auth: DEBUG: StringToSign: 2024-01-16T04:55:05.380 INFO:teuthology.orchestra.run.smithi156.stderr:AWS4-HMAC-SHA256 2024-01-16T04:55:05.380 INFO:teuthology.orchestra.run.smithi156.stderr:20240116T045210Z 2024-01-16T04:55:05.380 INFO:teuthology.orchestra.run.smithi156.stderr:20240116/us-east-1/s3/aws4_request 2024-01-16T04:55:05.380 INFO:teuthology.orchestra.run.smithi156.stderr:8f2844934c1a21a5482db36bd62678e1be9a86b0f0a013883520af63e9416405 2024-01-16T04:55:05.380 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.auth: DEBUG: Signature: 2024-01-16T04:55:05.380 INFO:teuthology.orchestra.run.smithi156.stderr:f254c813d3e83314711cf0704cce7c74b9ec9e6eaf9a57a56f68a8d4640bdcb1 2024-01-16T04:55:05.380 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2024-01-16T04:55:05.381 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.endpoint: DEBUG: Sending http request: 2024-01-16T04:55:05.381 INFO:teuthology.orchestra.run.smithi156.stderr:urllib3.connectionpool: DEBUG: http://smithi156.front.sepia.ceph.com:80 "PUT /test-client.0-kjdceoogenvalx4-212 HTTP/1.1" 200 0 2024-01-16T04:55:05.381 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx0000006c4b7e59ef7e16e-0065a60b7a-1061-default', 'Content-Length': '0', 'Date': 'Tue, 16 Jan 2024 04:52:10 GMT'} 2024-01-16T04:55:05.381 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.parsers: DEBUG: Response body: 2024-01-16T04:55:05.381 INFO:teuthology.orchestra.run.smithi156.stderr:b'' 2024-01-16T04:55:05.381 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2024-01-16T04:55:05.381 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.retryhandler: DEBUG: No retry needed. 2024-01-16T04:55:05.381 INFO:teuthology.orchestra.run.smithi156.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2024-01-16T04:55:05.381 INFO:teuthology.orchestra.run.smithi156.stderr:--------------------- >> end captured logging << --------------------- 2024-01-16T04:55:05.381 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:55:05.381 INFO:teuthology.orchestra.run.smithi156.stderr:---------------------------------------------------------------------- 2024-01-16T04:55:05.381 INFO:teuthology.orchestra.run.smithi156.stderr:Ran 566 tests in 209.023s 2024-01-16T04:55:05.381 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:55:05.381 INFO:teuthology.orchestra.run.smithi156.stderr:FAILED (SKIP=76, failures=2) 2024-01-16T04:55:05.438 INFO:teuthology.orchestra.run.smithi156.stderr:Exception ignored in: .generate at 0x7f8a27f14a40> 2024-01-16T04:55:05.438 INFO:teuthology.orchestra.run.smithi156.stderr:RuntimeError: generator ignored GeneratorExit 2024-01-16T04:55:05.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T04:55:05.472 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 441, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi156 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain'" 2024-01-16T04:55:05.474 INFO:tasks.s3tests:Cleaning up boto... 2024-01-16T04:55:05.475 DEBUG:teuthology.orchestra.run.smithi156:> rm /home/ubuntu/cephtest/boto.cfg 2024-01-16T04:55:05.494 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2024-01-16T04:55:05.574 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:55:05.574 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:55:05.581 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.583+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.584+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.584+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.584+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.584+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.585+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.585+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.585+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:55:05.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.592+0000 7ffb8f4ff7c0 20 RGWRados::pool_iterate: got zonegroup_info.92d3822f-5de2-431f-9d8f-b46ea51d44b6 2024-01-16T04:55:05.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.592+0000 7ffb8f4ff7c0 20 RGWRados::pool_iterate: got zone_info.2259a717-7b4f-4cc8-b097-8339a13202a0 2024-01-16T04:55:05.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.592+0000 7ffb8f4ff7c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:55:05.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.592+0000 7ffb8f4ff7c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:55:05.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.592+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.593+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:55:05.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.593+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.593+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:55:05.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.593+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.593+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:55:05.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.593+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.594+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:55:05.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.594+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.594+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.594+0000 7ffb8f4ff7c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:55:05.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.594+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.594+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.594+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.594+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:55:05.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.594+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.595+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:55:05.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.595+0000 7ffb8f4ff7c0 20 zonegroup default 2024-01-16T04:55:05.593 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.595+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.595+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.595+0000 7ffb8f4ff7c0 10 Cannot find current period zone using local zone 2024-01-16T04:55:05.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.595+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.595+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.595+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.595+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:55:05.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.595+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.595+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:55:05.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.595+0000 7ffb8f4ff7c0 20 zone default found 2024-01-16T04:55:05.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.595+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.596+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.596+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.596+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.596+0000 7ffb8f4ff7c0 20 started sync module instance, tier type = 2024-01-16T04:55:05.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.596+0000 7ffb8f4ff7c0 20 started zone id=2259a717-7b4f-4cc8-b097-8339a13202a0 (name=default) with tier type = 2024-01-16T04:55:05.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.603+0000 7ffb8f4ff7c0 20 add_watcher() i=0 2024-01-16T04:55:05.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.603+0000 7ffb8f4ff7c0 20 add_watcher() i=1 2024-01-16T04:55:05.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.603+0000 7ffb8f4ff7c0 20 add_watcher() i=2 2024-01-16T04:55:05.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.603+0000 7ffb8f4ff7c0 20 add_watcher() i=3 2024-01-16T04:55:05.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.603+0000 7ffb8f4ff7c0 20 add_watcher() i=4 2024-01-16T04:55:05.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.603+0000 7ffb8f4ff7c0 20 add_watcher() i=5 2024-01-16T04:55:05.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.603+0000 7ffb8f4ff7c0 20 add_watcher() i=6 2024-01-16T04:55:05.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.604+0000 7ffb8f4ff7c0 20 add_watcher() i=7 2024-01-16T04:55:05.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.604+0000 7ffb8f4ff7c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:55:05.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.606+0000 7ffb4bfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:55:05.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.606+0000 7ffb8f4ff7c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:55:05.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.606+0000 7ffb8f4ff7c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:55:05.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.643+0000 7ffabaffd700 20 reqs_thread_entry: start 2024-01-16T04:55:05.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.643+0000 7ffab8ff9700 20 reqs_thread_entry: start 2024-01-16T04:55:05.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.643+0000 7ffb8f4ff7c0 20 init_complete bucket index max shards: 11 2024-01-16T04:55:05.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.643+0000 7ffb8f4ff7c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:55:05.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.643+0000 7ffaaeffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:55:35 2024 2024-01-16T04:55:05.642 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:55:05.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.643+0000 7ffaae7fc700 20 reqs_thread_entry: start 2024-01-16T04:55:05.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.643+0000 7ffb8f4ff7c0 20 get_system_obj_state: rctx=0x7ffed4f3f9e8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55b81c178d50 s->prefetch_data=0 2024-01-16T04:55:05.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.643+0000 7ffb8f4ff7c0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2024-01-16T04:55:05.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.644+0000 7ffb8f4ff7c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2024-01-16T04:55:05.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.644+0000 7ffb8f4ff7c0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:55:05.642 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.644+0000 7ffb8f4ff7c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:55:05.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.644+0000 7ffb8f4ff7c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:55:05.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.644+0000 7ffb8f4ff7c0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:55:05.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.644+0000 7ffb8f4ff7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.644+0000 7ffb8f4ff7c0 20 rados_obj.operate() r=0 bl.length=469 2024-01-16T04:55:05.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.644+0000 7ffb8f4ff7c0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x11 2024-01-16T04:55:05.643 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.644+0000 7ffb8f4ff7c0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2024-01-16T04:55:05.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.645+0000 7ffb8f4ff7c0 10 removing key index: JCZHIUSFADNDMNFEHUYO 2024-01-16T04:55:05.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.645+0000 7ffb8f4ff7c0 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.keys:JCZHIUSFADNDMNFEHUYO, ofs0, ns] 2024-01-16T04:55:05.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.645+0000 7ffb6bfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823721 cookie 94249234975744 notifier 4235 bl.length()=170 2024-01-16T04:55:05.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.646+0000 7ffb8f4ff7c0 10 removing email index: foo.client.0+test@test.test 2024-01-16T04:55:05.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.646+0000 7ffb8f4ff7c0 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2024-01-16T04:55:05.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.646+0000 7ffb6b7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823722 cookie 94249234975744 notifier 4235 bl.length()=178 2024-01-16T04:55:05.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.648+0000 7ffb8f4ff7c0 10 removing user buckets index 2024-01-16T04:55:05.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.648+0000 7ffb8f4ff7c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2024-01-16T04:55:05.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.648+0000 7ffb6bfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823723 cookie 94249234968720 notifier 4235 bl.length()=169 2024-01-16T04:55:05.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.650+0000 7ffb8f4ff7c0 10 removing user index: foo.client.0 2024-01-16T04:55:05.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.650+0000 7ffb8f4ff7c0 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2024-01-16T04:55:05.647 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.650+0000 7ffb8f4ff7c0 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2024-01-16T04:55:05.648 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.650+0000 7ffb6b7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618822947 cookie 94249234966432 notifier 4235 bl.length()=161 2024-01-16T04:55:05.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.653+0000 7ffb8f4ff7c0 20 remove_watcher() i=0 2024-01-16T04:55:05.650 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.653+0000 7ffb8f4ff7c0 2 removed watcher, disabling cache 2024-01-16T04:55:05.651 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.654+0000 7ffb8f4ff7c0 20 remove_watcher() i=1 2024-01-16T04:55:05.652 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.654+0000 7ffb8f4ff7c0 20 remove_watcher() i=2 2024-01-16T04:55:05.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.655+0000 7ffb8f4ff7c0 20 remove_watcher() i=3 2024-01-16T04:55:05.653 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.656+0000 7ffb8f4ff7c0 20 remove_watcher() i=4 2024-01-16T04:55:05.654 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.657+0000 7ffb8f4ff7c0 20 remove_watcher() i=5 2024-01-16T04:55:05.655 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.658+0000 7ffb8f4ff7c0 20 remove_watcher() i=6 2024-01-16T04:55:05.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.659+0000 7ffb8f4ff7c0 20 remove_watcher() i=7 2024-01-16T04:55:05.656 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.659+0000 7ffb4bfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:55:05.661 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2024-01-16T04:55:05.742 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:55:05.742 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:55:05.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.750+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.751+0000 7f8576d2a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.751+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.751+0000 7f8576d2a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.751+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.751+0000 7f8576d2a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.751+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.751+0000 7f8576d2a7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:55:05.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.759+0000 7f8576d2a7c0 20 RGWRados::pool_iterate: got zonegroup_info.92d3822f-5de2-431f-9d8f-b46ea51d44b6 2024-01-16T04:55:05.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.759+0000 7f8576d2a7c0 20 RGWRados::pool_iterate: got zone_info.2259a717-7b4f-4cc8-b097-8339a13202a0 2024-01-16T04:55:05.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.759+0000 7f8576d2a7c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:55:05.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.759+0000 7f8576d2a7c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:55:05.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.759+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.759+0000 7f8576d2a7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:55:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.759+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.759+0000 7f8576d2a7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:55:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.759+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.759+0000 7f8576d2a7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:55:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.759+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.760+0000 7f8576d2a7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:55:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.760+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.760+0000 7f8576d2a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.760+0000 7f8576d2a7c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:55:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.760+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.760+0000 7f8576d2a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.760+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.760+0000 7f8576d2a7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:55:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.760+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.760+0000 7f8576d2a7c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:55:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.760+0000 7f8576d2a7c0 20 zonegroup default 2024-01-16T04:55:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.760+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.761+0000 7f8576d2a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.761+0000 7f8576d2a7c0 10 Cannot find current period zone using local zone 2024-01-16T04:55:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.761+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.761+0000 7f8576d2a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.761+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.761+0000 7f8576d2a7c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:55:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.761+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.761+0000 7f8576d2a7c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:55:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.761+0000 7f8576d2a7c0 20 zone default found 2024-01-16T04:55:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.761+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.761+0000 7f8576d2a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.761+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.762+0000 7f8576d2a7c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.762+0000 7f8576d2a7c0 20 started sync module instance, tier type = 2024-01-16T04:55:05.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.762+0000 7f8576d2a7c0 20 started zone id=2259a717-7b4f-4cc8-b097-8339a13202a0 (name=default) with tier type = 2024-01-16T04:55:05.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.769+0000 7f8576d2a7c0 20 add_watcher() i=0 2024-01-16T04:55:05.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.769+0000 7f8576d2a7c0 20 add_watcher() i=1 2024-01-16T04:55:05.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.769+0000 7f8576d2a7c0 20 add_watcher() i=2 2024-01-16T04:55:05.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.769+0000 7f8576d2a7c0 20 add_watcher() i=3 2024-01-16T04:55:05.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.769+0000 7f8576d2a7c0 20 add_watcher() i=4 2024-01-16T04:55:05.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.769+0000 7f8576d2a7c0 20 add_watcher() i=5 2024-01-16T04:55:05.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.769+0000 7f8576d2a7c0 20 add_watcher() i=6 2024-01-16T04:55:05.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.770+0000 7f8576d2a7c0 20 add_watcher() i=7 2024-01-16T04:55:05.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.770+0000 7f8576d2a7c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:55:05.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.771+0000 7f85337de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:55:05.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.771+0000 7f8576d2a7c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:55:05.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.771+0000 7f8576d2a7c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:55:05.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.809+0000 7f84a27fc700 20 reqs_thread_entry: start 2024-01-16T04:55:05.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.809+0000 7f8493fff700 20 reqs_thread_entry: start 2024-01-16T04:55:05.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.809+0000 7f8576d2a7c0 20 init_complete bucket index max shards: 11 2024-01-16T04:55:05.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.809+0000 7f8576d2a7c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:55:05.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.809+0000 7f84927fc700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:55:36 2024 2024-01-16T04:55:05.809 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:55:05.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.809+0000 7f8491ffb700 20 reqs_thread_entry: start 2024-01-16T04:55:05.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.809+0000 7f8576d2a7c0 20 get_system_obj_state: rctx=0x7ffc2e3109b8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x556358b37d50 s->prefetch_data=0 2024-01-16T04:55:05.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.809+0000 7f8576d2a7c0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2024-01-16T04:55:05.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.810+0000 7f8576d2a7c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2024-01-16T04:55:05.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.810+0000 7f8576d2a7c0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:55:05.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.810+0000 7f8576d2a7c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:55:05.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.810+0000 7f8576d2a7c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:55:05.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.810+0000 7f8576d2a7c0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:55:05.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.810+0000 7f8576d2a7c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.810+0000 7f8576d2a7c0 20 rados_obj.operate() r=0 bl.length=469 2024-01-16T04:55:05.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.810+0000 7f8576d2a7c0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x11 2024-01-16T04:55:05.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.810+0000 7f8576d2a7c0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2024-01-16T04:55:05.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.810+0000 7f8576d2a7c0 10 removing key index: QNZSUOJCMRFICNYBWQGD 2024-01-16T04:55:05.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.810+0000 7f8576d2a7c0 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:users.keys:QNZSUOJCMRFICNYBWQGD, ofs0, ns] 2024-01-16T04:55:05.810 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.811+0000 7f85537fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824458 cookie 93885179600928 notifier 4241 bl.length()=170 2024-01-16T04:55:05.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.812+0000 7f8576d2a7c0 10 removing email index: bar.client.0+test@test.test 2024-01-16T04:55:05.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.812+0000 7f8576d2a7c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2024-01-16T04:55:05.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.812+0000 7f8552ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823724 cookie 93885179597440 notifier 4241 bl.length()=178 2024-01-16T04:55:05.811 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.813+0000 7f8576d2a7c0 10 removing user buckets index 2024-01-16T04:55:05.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.813+0000 7f8576d2a7c0 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0.buckets, ofs0, ns] 2024-01-16T04:55:05.812 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.814+0000 7f85537fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824459 cookie 93885179599824 notifier 4241 bl.length()=169 2024-01-16T04:55:05.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.815+0000 7f8576d2a7c0 10 removing user index: bar.client.0 2024-01-16T04:55:05.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.815+0000 7f8576d2a7c0 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2024-01-16T04:55:05.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.815+0000 7f8576d2a7c0 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2024-01-16T04:55:05.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.815+0000 7f8552ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823725 cookie 93885179597440 notifier 4241 bl.length()=161 2024-01-16T04:55:05.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.818+0000 7f8576d2a7c0 20 remove_watcher() i=0 2024-01-16T04:55:05.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.818+0000 7f8576d2a7c0 2 removed watcher, disabling cache 2024-01-16T04:55:05.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.819+0000 7f8576d2a7c0 20 remove_watcher() i=1 2024-01-16T04:55:05.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.819+0000 7f8576d2a7c0 20 remove_watcher() i=2 2024-01-16T04:55:05.817 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.820+0000 7f8576d2a7c0 20 remove_watcher() i=3 2024-01-16T04:55:05.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.821+0000 7f8576d2a7c0 20 remove_watcher() i=4 2024-01-16T04:55:05.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.822+0000 7f8576d2a7c0 20 remove_watcher() i=5 2024-01-16T04:55:05.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.823+0000 7f8576d2a7c0 20 remove_watcher() i=6 2024-01-16T04:55:05.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.823+0000 7f8576d2a7c0 20 remove_watcher() i=7 2024-01-16T04:55:05.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.823+0000 7f85337de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:55:05.826 DEBUG:teuthology.orchestra.run.smithi156:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2024-01-16T04:55:05.906 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setuser ceph since I am not root 2024-01-16T04:55:05.906 INFO:teuthology.orchestra.run.smithi156.stderr:ignoring --setgroup ceph since I am not root 2024-01-16T04:55:05.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.914+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.915+0000 7fec8ac157c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.912 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.915+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.915+0000 7fec8ac157c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.915+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.915+0000 7fec8ac157c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.915+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.913 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.915+0000 7fec8ac157c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:55:05.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.922+0000 7fec8ac157c0 20 RGWRados::pool_iterate: got zonegroup_info.92d3822f-5de2-431f-9d8f-b46ea51d44b6 2024-01-16T04:55:05.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.922+0000 7fec8ac157c0 20 RGWRados::pool_iterate: got zone_info.2259a717-7b4f-4cc8-b097-8339a13202a0 2024-01-16T04:55:05.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.922+0000 7fec8ac157c0 20 RGWRados::pool_iterate: got zone_names.default 2024-01-16T04:55:05.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.922+0000 7fec8ac157c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2024-01-16T04:55:05.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.922+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.922+0000 7fec8ac157c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:55:05.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.922+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.923+0000 7fec8ac157c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:55:05.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.923+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.923+0000 7fec8ac157c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:55:05.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.923+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.923+0000 7fec8ac157c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:55:05.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.923+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.923+0000 7fec8ac157c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.923+0000 7fec8ac157c0 10 cannot find current period zonegroup using local zonegroup 2024-01-16T04:55:05.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.923+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.923+0000 7fec8ac157c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.923+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.924+0000 7fec8ac157c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:55:05.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.924+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.924+0000 7fec8ac157c0 20 rados_obj.operate() r=0 bl.length=358 2024-01-16T04:55:05.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.924+0000 7fec8ac157c0 20 zonegroup default 2024-01-16T04:55:05.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.924+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.924+0000 7fec8ac157c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.924+0000 7fec8ac157c0 10 Cannot find current period zone using local zone 2024-01-16T04:55:05.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.924+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.924+0000 7fec8ac157c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.924+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.925+0000 7fec8ac157c0 20 rados_obj.operate() r=0 bl.length=46 2024-01-16T04:55:05.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.925+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.925+0000 7fec8ac157c0 20 rados_obj.operate() r=0 bl.length=873 2024-01-16T04:55:05.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.925+0000 7fec8ac157c0 20 zone default found 2024-01-16T04:55:05.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.925+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.925+0000 7fec8ac157c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.925+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.925+0000 7fec8ac157c0 20 rados_obj.operate() r=-2 bl.length=0 2024-01-16T04:55:05.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.925+0000 7fec8ac157c0 20 started sync module instance, tier type = 2024-01-16T04:55:05.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.925+0000 7fec8ac157c0 20 started zone id=2259a717-7b4f-4cc8-b097-8339a13202a0 (name=default) with tier type = 2024-01-16T04:55:05.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.933+0000 7fec8ac157c0 20 add_watcher() i=0 2024-01-16T04:55:05.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.933+0000 7fec8ac157c0 20 add_watcher() i=1 2024-01-16T04:55:05.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.933+0000 7fec8ac157c0 20 add_watcher() i=2 2024-01-16T04:55:05.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.933+0000 7fec8ac157c0 20 add_watcher() i=3 2024-01-16T04:55:05.930 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.933+0000 7fec8ac157c0 20 add_watcher() i=4 2024-01-16T04:55:05.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.933+0000 7fec8ac157c0 20 add_watcher() i=5 2024-01-16T04:55:05.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.933+0000 7fec8ac157c0 20 add_watcher() i=6 2024-01-16T04:55:05.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.933+0000 7fec8ac157c0 20 add_watcher() i=7 2024-01-16T04:55:05.931 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.933+0000 7fec8ac157c0 2 all 8 watchers are set, enabling cache 2024-01-16T04:55:05.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.935+0000 7fec477de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:55:05.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.935+0000 7fec8ac157c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2024-01-16T04:55:05.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.935+0000 7fec8ac157c0 20 check_secure_mon_conn(): mode 1 is insecure 2024-01-16T04:55:05.972 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.973+0000 7febb67fc700 20 reqs_thread_entry: start 2024-01-16T04:55:05.972 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.973+0000 7feba7fff700 20 reqs_thread_entry: start 2024-01-16T04:55:05.972 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.973+0000 7fec8ac157c0 20 init_complete bucket index max shards: 11 2024-01-16T04:55:05.973 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.973+0000 7fec8ac157c0 10 rgw notify: Started notification manager with: 1 workers 2024-01-16T04:55:05.973 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.973+0000 7feba67fc700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 16 04:55:36 2024 2024-01-16T04:55:05.973 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:55:05.973 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.973+0000 7feba5ffb700 20 reqs_thread_entry: start 2024-01-16T04:55:05.973 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.974+0000 7fec8ac157c0 20 get_system_obj_state: rctx=0x7ffd9425ab48 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55f57457dd50 s->prefetch_data=0 2024-01-16T04:55:05.973 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.974+0000 7fec8ac157c0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2024-01-16T04:55:05.973 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.974+0000 7fec8ac157c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2024-01-16T04:55:05.973 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.974+0000 7fec8ac157c0 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:55:05.973 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.974+0000 7fec8ac157c0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2024-01-16T04:55:05.973 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.974+0000 7fec8ac157c0 10 updating xattr: name=user.rgw.user-policy bl.length()=118 2024-01-16T04:55:05.973 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.974+0000 7fec8ac157c0 20 get_system_obj_state: s->obj_tag was set empty 2024-01-16T04:55:05.974 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.974+0000 7fec8ac157c0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x11, cached=0x16) 2024-01-16T04:55:05.974 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.974+0000 7fec8ac157c0 20 rados->read ofs=0 len=0 2024-01-16T04:55:05.974 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.974+0000 7fec8ac157c0 20 rados_obj.operate() r=0 bl.length=528 2024-01-16T04:55:05.974 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.974+0000 7fec8ac157c0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x11 2024-01-16T04:55:05.974 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.974+0000 7fec8ac157c0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2024-01-16T04:55:05.974 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.975+0000 7fec8ac157c0 10 removing key index: WFOUTFGFFAJQEWTWXRIU 2024-01-16T04:55:05.974 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.975+0000 7fec8ac157c0 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:users.keys:WFOUTFGFFAJQEWTWXRIU, ofs0, ns] 2024-01-16T04:55:05.974 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.975+0000 7fec677fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618824460 cookie 94512708587280 notifier 4247 bl.length()=170 2024-01-16T04:55:05.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.976+0000 7fec8ac157c0 10 removing email index: testx$tenanteduser.client.0+test@test.test 2024-01-16T04:55:05.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.976+0000 7fec8ac157c0 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2024-01-16T04:55:05.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.976+0000 7fec677fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618822948 cookie 94512708596112 notifier 4247 bl.length()=193 2024-01-16T04:55:05.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.977+0000 7fec8ac157c0 10 removing user buckets index 2024-01-16T04:55:05.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.977+0000 7fec8ac157c0 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0.buckets, ofs0, ns] 2024-01-16T04:55:05.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.978+0000 7fec677fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618823726 cookie 94512708592448 notifier 4247 bl.length()=184 2024-01-16T04:55:05.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.979+0000 7fec8ac157c0 10 removing user index: testx$tenanteduser.client.0 2024-01-16T04:55:05.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.979+0000 7fec8ac157c0 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2024-01-16T04:55:05.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.979+0000 7fec8ac157c0 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2024-01-16T04:55:05.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.979+0000 7fec66ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618822949 cookie 94512708588464 notifier 4247 bl.length()=176 2024-01-16T04:55:05.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.982+0000 7fec8ac157c0 20 remove_watcher() i=0 2024-01-16T04:55:05.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.982+0000 7fec8ac157c0 2 removed watcher, disabling cache 2024-01-16T04:55:05.980 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.983+0000 7fec8ac157c0 20 remove_watcher() i=1 2024-01-16T04:55:05.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.984+0000 7fec8ac157c0 20 remove_watcher() i=2 2024-01-16T04:55:05.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.985+0000 7fec8ac157c0 20 remove_watcher() i=3 2024-01-16T04:55:05.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.985+0000 7fec8ac157c0 20 remove_watcher() i=4 2024-01-16T04:55:05.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.986+0000 7fec8ac157c0 20 remove_watcher() i=5 2024-01-16T04:55:05.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.987+0000 7fec8ac157c0 20 remove_watcher() i=6 2024-01-16T04:55:05.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.988+0000 7fec8ac157c0 20 remove_watcher() i=7 2024-01-16T04:55:05.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-01-16T04:55:05.988+0000 7fec477de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2024-01-16T04:55:05.990 INFO:tasks.s3tests:Removing s3-tests... 2024-01-16T04:55:05.990 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf /home/ubuntu/cephtest/s3-tests 2024-01-16T04:55:06.150 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 639, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 59, in download yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 441, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi156 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain'" 2024-01-16T04:55:06.401 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=19354cf28db14f418394516b84e438ff Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 639, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 59, in download yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/s3tests.py", line 441, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi156 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain'" 2024-01-16T04:55:06.405 DEBUG:teuthology.run_tasks:Unwinding manager s3tests 2024-01-16T04:55:06.415 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2024-01-16T04:55:06.440 DEBUG:tasks.rgw.client.0:waiting for process to exit 2024-01-16T04:55:06.440 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:55:06.445 INFO:tasks.rgw.client.0.smithi156.stdout:2024-01-16T04:55:06.444+0000 7f30ce695700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends civetweb port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_vault_addr smithi156.front.sepia.ceph.com:8200 --rgw_crypt_vault_token_file /etc/ceph/vault-root-token --foreground (PID: 43177) UID: 0 2024-01-16T04:55:06.445 INFO:tasks.rgw.client.0.smithi156.stdout:2024-01-16T04:55:06.444+0000 7f30e6391880 -1 shutting down 2024-01-16T04:55:09.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2024-01-16T04:55:12.542 INFO:tasks.rgw.client.0:Stopped 2024-01-16T04:55:12.543 DEBUG:teuthology.orchestra.run.smithi156:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2024-01-16T04:55:12.565 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/vault-root-token 2024-01-16T04:55:12.640 DEBUG:teuthology.run_tasks:Unwinding manager vault 2024-01-16T04:55:12.651 INFO:tasks.vault:Stopping Vault instance 2024-01-16T04:55:12.651 DEBUG:tasks.vault.client.0:waiting for process to exit 2024-01-16T04:55:12.651 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:55:12.681 INFO:tasks.vault.client.0.smithi156.stdout:==> Vault shutdown triggered 2024-01-16T04:55:12.682 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:55:12.685Z [INFO] core: marked as sealed 2024-01-16T04:55:12.682 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:55:12.685Z [INFO] core: pre-seal teardown starting 2024-01-16T04:55:12.682 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:55:12.685Z [INFO] rollback: stopping rollback manager 2024-01-16T04:55:12.683 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:55:12.685Z [INFO] core: pre-seal teardown complete 2024-01-16T04:55:12.683 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:55:12.685Z [INFO] core: stopping cluster listeners 2024-01-16T04:55:12.683 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:55:12.685Z [INFO] core.cluster-listener: forwarding rpc listeners stopped 2024-01-16T04:55:13.004 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:55:13.008Z [INFO] core.cluster-listener: rpc listeners successfully shut down 2024-01-16T04:55:13.005 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:55:13.008Z [INFO] core: cluster listeners successfully shut down 2024-01-16T04:55:13.005 INFO:tasks.vault.client.0.smithi156.stderr:2024-01-16T04:55:13.008Z [INFO] core: vault is sealed 2024-01-16T04:55:13.006 INFO:tasks.vault.client.0:Stopped 2024-01-16T04:55:13.006 INFO:tasks.vault:Removing Vault... 2024-01-16T04:55:13.006 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf /home/ubuntu/cephtest/vault /home/ubuntu/cephtest/vault.zip 2024-01-16T04:55:13.085 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-16T04:55:13.095 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-01-16T04:55:13.095 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T04:55:13.424 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-01-16T04:55:13.424 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":36,"fsid":"dae07de8-a7c7-4fa7-b3ca-19c915fbdf1d","created":"2024-01-16T04:50:07.150411+0000","modified":"2024-01-16T04:52:14.139412+0000","last_up_change":"2024-01-16T04:50:14.516683+0000","last_in_change":"2024-01-16T04:50:08.136013+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-01-16T04:50:13.551691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-16T04:50:15.831542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2024-01-16T04:50:34.732764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2024-01-16T04:50:36.490427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2024-01-16T04:50:38.628536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2024-01-16T04:50:40.222493+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2024-01-16T04:50:42.205316+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2024-01-16T04:50:44.260863+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"default.rgw.otp","create_time":"2024-01-16T04:50:54.612598+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"default.rgw.buckets.non-ec","create_time":"2024-01-16T04:51:37.484547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"2fd312a6-3d18-421f-b641-a81c9bb49196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":40342},{"type":"v1","addr":"172.21.15.156:6801","nonce":40342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":40342},{"type":"v1","addr":"172.21.15.156:6803","nonce":40342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":40342},{"type":"v1","addr":"172.21.15.156:6807","nonce":40342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":40342},{"type":"v1","addr":"172.21.15.156:6805","nonce":40342}]},"public_addr":"172.21.15.156:6801/40342","cluster_addr":"172.21.15.156:6803/40342","heartbeat_back_addr":"172.21.15.156:6807/40342","heartbeat_front_addr":"172.21.15.156:6805/40342","state":["exists","up"]},{"osd":1,"uuid":"9189293d-261d-4048-a9de-6a5778c6ce44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":40343},{"type":"v1","addr":"172.21.15.156:6809","nonce":40343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":40343},{"type":"v1","addr":"172.21.15.156:6811","nonce":40343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":40343},{"type":"v1","addr":"172.21.15.156:6815","nonce":40343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":40343},{"type":"v1","addr":"172.21.15.156:6813","nonce":40343}]},"public_addr":"172.21.15.156:6809/40343","cluster_addr":"172.21.15.156:6811/40343","heartbeat_back_addr":"172.21.15.156:6815/40343","heartbeat_front_addr":"172.21.15.156:6813/40343","state":["exists","up"]},{"osd":2,"uuid":"65c3a83b-fea3-406d-a04b-67e1cdd6856e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":40372},{"type":"v1","addr":"172.21.15.156:6817","nonce":40372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":40372},{"type":"v1","addr":"172.21.15.156:6819","nonce":40372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":40372},{"type":"v1","addr":"172.21.15.156:6823","nonce":40372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":40372},{"type":"v1","addr":"172.21.15.156:6821","nonce":40372}]},"public_addr":"172.21.15.156:6817/40372","cluster_addr":"172.21.15.156:6819/40372","heartbeat_back_addr":"172.21.15.156:6823/40372","heartbeat_front_addr":"172.21.15.156:6821/40372","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:50:12.447063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:50:12.508216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T04:50:12.525157+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.6","mappings":[{"from":0,"to":2}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":0}]},{"pgid":"4.12","mappings":[{"from":1,"to":0}]},{"pgid":"4.13","mappings":[{"from":1,"to":0}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]},{"pgid":"4.30","mappings":[{"from":1,"to":2}]},{"pgid":"4.33","mappings":[{"from":1,"to":0}]},{"pgid":"10.14","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T04:55:13.438 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-16T04:55:13.740 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-16T04:55:13.741 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-16T04:55:13.741 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-16T04:55:13.741 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:55:13.791 INFO:tasks.ceph.osd.0:Stopped 2024-01-16T04:55:13.792 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-16T04:55:13.792 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:55:13.848 INFO:tasks.ceph.osd.1:Stopped 2024-01-16T04:55:13.848 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-16T04:55:13.848 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:55:13.894 INFO:tasks.ceph.osd.2:Stopped 2024-01-16T04:55:13.894 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-16T04:55:13.894 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-16T04:55:13.894 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:55:13.915 INFO:tasks.ceph.mgr.x:Stopped 2024-01-16T04:55:13.915 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-16T04:55:13.916 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-16T04:55:13.916 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T04:55:13.966 INFO:tasks.ceph.mon.a:Stopped 2024-01-16T04:55:13.966 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-16T04:55:13.966 DEBUG:teuthology.orchestra.run.smithi156:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-01-16T04:55:14.034 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi156.front.sepia.ceph.com 2024-01-16T04:55:14.035 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-16T04:55:14.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2024-01-16T04:55:14.789 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi156.front.sepia.ceph.com 2024-01-16T04:55:14.790 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-16T04:55:14.873 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi156.front.sepia.ceph.com 2024-01-16T04:55:14.874 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-16T04:55:14.948 INFO:tasks.ceph:Archiving mon data... 2024-01-16T04:55:14.950 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517339/data/mon.a.tgz 2024-01-16T04:55:14.950 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2024-01-16T04:55:14.969 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.9glrCfFCLv 2024-01-16T04:55:14.969 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.9glrCfFCLv 2024-01-16T04:55:15.233 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.9glrCfFCLv 2024-01-16T04:55:15.321 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.9glrCfFCLv is 1003KB 2024-01-16T04:55:15.427 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.9glrCfFCLv 2024-01-16T04:55:15.445 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-16T04:55:15.445 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-16T04:55:15.511 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-01-16T04:55:15.511 DEBUG:teuthology.orchestra.run.smithi156:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-01-16T04:55:15.579 INFO:teuthology.orchestra.run.smithi156.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-01-16T04:55:15.584 INFO:tasks.ceph:Archiving crash dumps... 2024-01-16T04:55:15.586 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517339/remote/smithi156/crash 2024-01-16T04:55:15.587 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-16T04:55:15.650 INFO:tasks.ceph:Compressing logs... 2024-01-16T04:55:15.650 DEBUG:teuthology.orchestra.run.smithi156:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-16T04:55:15.725 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40168.log 2024-01-16T04:55:15.726 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42318.log 2024-01-16T04:55:15.726 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42514.log 2024-01-16T04:55:15.726 INFO:teuthology.orchestra.run.smithi156.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40168.log.gz 2024-01-16T04:55:15.726 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42612.log 2024-01-16T04:55:15.726 INFO:teuthology.orchestra.run.smithi156.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42318.log.gz 2024-01-16T04:55:15.726 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/rgw.ceph.client.0.log 2024-01-16T04:55:15.727 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42514.log.gz 2024-01-16T04:55:15.727 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46824.log 2024-01-16T04:55:15.727 INFO:teuthology.orchestra.run.smithi156.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42612.log.gz 2024-01-16T04:55:15.727 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42956.log 2024-01-16T04:55:15.727 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/rgw.ceph.client.0.log: /var/log/ceph/ceph-client.admin.46824.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-16T04:55:15.727 INFO:teuthology.orchestra.run.smithi156.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46824.log.gz 2024-01-16T04:55:15.727 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42956.log.gz 2024-01-16T04:55:15.727 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41975.log 2024-01-16T04:55:15.727 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46357.log 2024-01-16T04:55:15.728 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41975.log.gz 2024-01-16T04:55:15.728 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42563.log 2024-01-16T04:55:15.728 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42860.log 2024-01-16T04:55:15.728 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.0.46357.log: /var/log/ceph/ceph-client.admin.42563.log: 83.3% -- replaced with /var/log/ceph/ceph-client.0.46357.log.gz 2024-01-16T04:55:15.728 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.36408.log 2024-01-16T04:55:15.728 INFO:teuthology.orchestra.run.smithi156.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42563.log.gz 2024-01-16T04:55:15.728 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42367.log 2024-01-16T04:55:15.728 INFO:teuthology.orchestra.run.smithi156.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42860.log.gz 2024-01-16T04:55:15.729 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-16T04:55:15.729 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph.tmp-client.admin.36408.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.36408.log.gz 2024-01-16T04:55:15.729 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42367.log.gz 2024-01-16T04:55:15.729 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40414.log 2024-01-16T04:55:15.729 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44419.log 2024-01-16T04:55:15.729 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40414.log.gz 2024-01-16T04:55:15.729 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39933.log 2024-01-16T04:55:15.729 INFO:teuthology.orchestra.run.smithi156.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-16T04:55:15.730 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.0.44419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44125.log 2024-01-16T04:55:15.730 INFO:teuthology.orchestra.run.smithi156.stderr: 83.5% -- replaced with /var/log/ceph/ceph-client.0.44419.log.gz 2024-01-16T04:55:15.730 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.39933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44321.log 2024-01-16T04:55:15.730 INFO:teuthology.orchestra.run.smithi156.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39933.log.gz 2024-01-16T04:55:15.730 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.0.44125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42269.log 2024-01-16T04:55:15.730 INFO:teuthology.orchestra.run.smithi156.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.0.44125.log.gz 2024-01-16T04:55:15.730 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.0.44321.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-16T04:55:15.730 INFO:teuthology.orchestra.run.smithi156.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.0.44321.log.gz 2024-01-16T04:55:15.730 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42465.log 2024-01-16T04:55:15.731 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42269.log.gz 2024-01-16T04:55:15.731 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42122.log 2024-01-16T04:55:15.731 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42073.log 2024-01-16T04:55:15.731 INFO:teuthology.orchestra.run.smithi156.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42465.log.gz 2024-01-16T04:55:15.731 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42122.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.42122.log.gz-- 2024-01-16T04:55:15.731 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42661.log 2024-01-16T04:55:15.731 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-16T04:55:15.731 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42073.log.gz 2024-01-16T04:55:15.731 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46777.log 2024-01-16T04:55:15.732 INFO:teuthology.orchestra.run.smithi156.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42661.log.gz 2024-01-16T04:55:15.732 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42416.log 2024-01-16T04:55:15.732 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.46777.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46777.log.gz 2024-01-16T04:55:15.732 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42416.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42416.log.gz 2024-01-16T04:55:15.732 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46455.log 2024-01-16T04:55:15.732 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43927.log 2024-01-16T04:55:15.737 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.0.46455.log: 83.4% -- replaced with /var/log/ceph/ceph-client.0.46455.log.gz 2024-01-16T04:55:15.737 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.0.43927.log: 86.1% -- replaced with /var/log/ceph/ceph-client.0.43927.log.gz 2024-01-16T04:55:15.737 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-16T04:55:15.737 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43005.log 2024-01-16T04:55:15.737 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.43005.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43005.log.gz 2024-01-16T04:55:15.737 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40215.log 2024-01-16T04:55:15.737 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42909.log 2024-01-16T04:55:15.738 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40215.log.gz 2024-01-16T04:55:15.738 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42909.log.gz 2024-01-16T04:55:15.738 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-16T04:55:15.738 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40119.log 2024-01-16T04:55:15.738 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.40119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40119.log.gz 2024-01-16T04:55:15.738 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42024.log 2024-01-16T04:55:15.738 INFO:teuthology.orchestra.run.smithi156.stderr: 85.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-16T04:55:15.738 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42024.log.gz 2024-01-16T04:55:15.738 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41906.log 2024-01-16T04:55:15.739 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42220.log 2024-01-16T04:55:15.739 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.41906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41906.log.gz 2024-01-16T04:55:15.739 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42220.log.gz 2024-01-16T04:55:15.739 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44025.log 2024-01-16T04:55:15.739 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44223.log 2024-01-16T04:55:15.739 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.0.44025.log: 83.2% -- replaced with /var/log/ceph/ceph-client.0.44025.log.gz 2024-01-16T04:55:15.739 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.0.44223.log: 83.3% -- replaced with /var/log/ceph/ceph-client.0.44223.log.gz 2024-01-16T04:55:15.739 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40341.log 2024-01-16T04:55:15.740 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42171.log 2024-01-16T04:55:15.740 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.40341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40341.log.gz 2024-01-16T04:55:15.740 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.42171.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42171.log.gz 2024-01-16T04:55:15.740 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-16T04:55:15.740 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46553.log 2024-01-16T04:55:15.740 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.0.46553.log: 83.5% -- replaced with /var/log/ceph/ceph-client.0.46553.log.gz 2024-01-16T04:55:15.756 INFO:teuthology.orchestra.run.smithi156.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-16T04:55:15.848 INFO:teuthology.orchestra.run.smithi156.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-16T04:55:16.861 INFO:teuthology.orchestra.run.smithi156.stderr: 93.9% -- replaced with /var/log/ceph/rgw.ceph.client.0.log.gz 2024-01-16T04:55:20.602 INFO:teuthology.orchestra.run.smithi156.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-16T04:55:22.237 INFO:teuthology.orchestra.run.smithi156.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-16T04:55:22.343 INFO:teuthology.orchestra.run.smithi156.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-16T04:55:22.344 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-01-16T04:55:22.345 INFO:teuthology.orchestra.run.smithi156.stderr:real 0m6.637s 2024-01-16T04:55:22.345 INFO:teuthology.orchestra.run.smithi156.stderr:user 0m18.569s 2024-01-16T04:55:22.345 INFO:teuthology.orchestra.run.smithi156.stderr:sys 0m0.755s 2024-01-16T04:55:22.345 INFO:tasks.ceph:Archiving logs... 2024-01-16T04:55:22.345 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517339/remote/smithi156/log 2024-01-16T04:55:22.346 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-16T04:55:25.134 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-16T04:55:25.144 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-16T04:55:25.144 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-16T04:55:25.212 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-16T04:55:25.220 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-16T04:55:25.220 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T04:55:25.245 INFO:teuthology.orchestra.run.smithi156.stderr:bash: ntpq: command not found 2024-01-16T04:55:25.250 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T04:55:25.250 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-01-16T04:55:25.250 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +2785us[+2785us] +/- 86ms 2024-01-16T04:55:25.250 INFO:teuthology.orchestra.run.smithi156.stdout:^* hv02.front.sepia.ceph.com 3 6 377 26 -273us[ -272us] +/- 79ms 2024-01-16T04:55:25.250 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 -2808us[-2808us] +/- 96ms 2024-01-16T04:55:25.251 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T04:55:25.251 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-16T04:55:25.260 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-16T04:55:25.260 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-16T04:55:25.269 DEBUG:teuthology.orchestra.run.smithi156:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-16T04:55:25.359 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T04:55:25.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T04:55:25.390 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-16T04:55:25.402 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-16T04:55:25.417 INFO:teuthology.task.internal:Duration was 897.533314 seconds 2024-01-16T04:55:25.417 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-16T04:55:25.429 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-16T04:55:25.429 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-16T04:55:25.478 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T04:55:25.859 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-16T04:55:25.860 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2024-01-16T04:55:25.860 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-16T04:55:25.883 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-16T04:55:25.883 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-16T04:55:25.970 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-16T04:55:25.970 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-16T04:55:26.077 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-16T04:55:26.089 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-16T04:55:26.089 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-16T04:55:26.116 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-16T04:55:26.130 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-16T04:55:26.183 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2024-01-16T04:55:26.206 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-16T04:55:26.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T04:55:26.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-16T04:55:26.274 INFO:teuthology.task.internal:Transferring archived files... 2024-01-16T04:55:26.274 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517339/remote/smithi156 2024-01-16T04:55:26.274 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-16T04:55:26.344 INFO:teuthology.task.internal:Removing archive directory... 2024-01-16T04:55:26.344 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-16T04:55:26.394 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-16T04:55:26.407 INFO:teuthology.task.internal:Not uploading archives. 2024-01-16T04:55:26.407 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-16T04:55:26.422 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-16T04:55:26.423 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-16T04:55:26.451 INFO:teuthology.orchestra.run.smithi156.stdout: 526082 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 16 04:55 /home/ubuntu/cephtest 2024-01-16T04:55:26.453 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-16T04:55:26.467 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-16T04:55:26.495 INFO:teuthology.nuke:Checking targets against current locks 2024-01-16T04:55:26.521 DEBUG:teuthology.nuke:shortname: smithi156 2024-01-16T04:55:26.522 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T04:55:26.548 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-15_22:05:22-rgw-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7517339', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 04:32:17.387869', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNs8qpHHPPUS2s7++BnIXXAgH3hvybrKv4GM3m7Zbb48nI7wNNZl355X3XAuPc/Bw79V8QAbjDfp0cJPFstEVA4='} 2024-01-16T04:55:26.604 INFO:teuthology.orchestra.console.smithi156:Power off 2024-01-16T04:55:26.605 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-16T04:55:26.629 DEBUG:teuthology.orchestra.console.smithi156:power off output: Chassis Power Control: Down/Off 2024-01-16T04:55:26.629 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T04:55:26.653 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2024-01-16T04:55:30.655 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T04:55:30.767 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2024-01-16T04:55:34.768 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T04:55:34.878 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off 2024-01-16T04:55:34.979 INFO:teuthology.orchestra.console.smithi156:Power off completed 2024-01-16T04:55:35.084 INFO:teuthology.run:Summary data: description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} duration: 897.5333139896393 failure_reason: 'Command failed (s3 tests against rgw) on smithi156 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a ''!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain''"' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=19354cf28db14f418394516b84e438ff status: fail success: false 2024-01-16T04:55:35.085 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T04:55:35.160 INFO:teuthology.run:FAIL