2023-12-07T21:44:17.405 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-07T21:44:17.405 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482401 --owner scheduled_yuriw@teuthology --description rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} --name yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi --verbose -- /home/teuthworker/archive/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482401/orig.config.yaml 2023-12-07T21:44:17.435 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-07T21:44:17.509 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482401 branch: wip-yuri5-testing-2023-12-07-0740-pacific description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} email: yweinste@redhat.com first_in_suite: false job_id: '7482401' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri5-testing-2023-12-07-0740-pacific ceph: conf: client: debug rgw: 20 rgw crypt kmip ca path: /etc/ceph/kmiproot.crt rgw crypt kmip client cert: /etc/ceph/kmip-client.crt rgw crypt kmip client key: /etc/ceph/kmip-client.key rgw crypt kmip kms key template: pykmip-$keyid rgw crypt require ssl: false rgw crypt s3 kms backend: kmip setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(POOL_APP_NOT_ENABLED\) sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: default sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 rgw: client.0: use-pykmip-role: client.0 workunit: branch: wip-yuri5-testing-2023-12-07-0740-pacific sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 616 sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 sleep_before_teardown: 0 subset: 111/12000 suite: rgw suite_branch: wip-yuri5-testing-2023-12-07-0740-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 targets: smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - openssl_keys: kmip-client: ca: kmiproot client: client.0 cn: rgw-client kmip-server: ca: kmiproot client: client.0 kmiproot: client: client.0 cn: kmiproot key-type: rsa:4096 - exec: client.0: - chmod 644 /home/ubuntu/cephtest/ca/kmip-client.key - pykmip: client.0: clientca: kmiproot clientcert: kmip-client secrets: - name: pykmip-my-key-1 - name: pykmip-my-key-2 servercert: kmip-server - rgw: client.0: null - s3tests: client.0: barbican: kms_key: my-key-1 kms_key2: my-key-2 force-branch: ceph-pacific vault_kv: key_path: my-key-1 key_path2: my-key-2 vault_old: key_path: my-key-1/1 key_path2: my-key-2/1 vault_transit: key_path: my-key-1 key_path2: my-key-2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-07_21:36:14 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-07T21:44:17.509 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa; will attempt to use it 2023-12-07T21:44:17.509 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks 2023-12-07T21:44:17.509 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-07T21:44:17.511 INFO:teuthology.task.internal:Checking packages... 2023-12-07T21:44:17.531 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '451ceae11b5d4caee05e94372fe658621eefc0b1' 2023-12-07T21:44:17.531 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-07T21:44:17.532 INFO:teuthology.packaging:ref: None 2023-12-07T21:44:17.532 INFO:teuthology.packaging:tag: None 2023-12-07T21:44:17.532 INFO:teuthology.packaging:branch: wip-yuri5-testing-2023-12-07-0740-pacific 2023-12-07T21:44:17.532 INFO:teuthology.packaging:sha1: 451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-07T21:44:17.532 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri5-testing-2023-12-07-0740-pacific 2023-12-07T21:44:17.677 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-365-g451ceae1-1focal 2023-12-07T21:44:17.677 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-07T21:44:17.684 INFO:teuthology.task.internal:no buildpackages task found 2023-12-07T21:44:17.684 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-07T21:44:17.824 INFO:teuthology.task.internal:Saving configuration 2023-12-07T21:44:17.836 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-07T21:44:17.867 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-07T21:44:17.890 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482401', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-07 21:38:41.080999', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-12-07T21:44:17.891 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-07T21:44:17.897 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-12-07T21:44:17.897 INFO:teuthology.run_tasks:Running task console_log... 2023-12-07T21:44:17.949 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe2e4109ee0>, signals=[15]) 2023-12-07T21:44:17.949 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-07T21:44:17.955 INFO:teuthology.task.internal:Opening connections... 2023-12-07T21:44:17.955 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2023-12-07T21:44:17.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T21:44:18.001 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-07T21:44:18.007 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2023-12-07T21:44:18.133 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2023-12-07T21:44:18.133 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2023-12-07T21:44:18.186 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="Ubuntu" 2023-12-07T21:44:18.186 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-12-07T21:44:18.186 INFO:teuthology.orchestra.run.smithi033.stdout:ID=ubuntu 2023-12-07T21:44:18.186 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE=debian 2023-12-07T21:44:18.186 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-12-07T21:44:18.186 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="20.04" 2023-12-07T21:44:18.186 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-07T21:44:18.186 INFO:teuthology.orchestra.run.smithi033.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-07T21:44:18.186 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-07T21:44:18.187 INFO:teuthology.orchestra.run.smithi033.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-07T21:44:18.187 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_CODENAME=focal 2023-12-07T21:44:18.187 INFO:teuthology.orchestra.run.smithi033.stdout:UBUNTU_CODENAME=focal 2023-12-07T21:44:18.187 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2023-12-07T21:44:18.205 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-07T21:44:18.213 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-07T21:44:18.240 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-07T21:44:18.240 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2023-12-07T21:44:18.246 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-07T21:44:18.260 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-07T21:44:18.260 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2023-12-07T21:44:18.269 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-07T21:44:18.300 INFO:teuthology.run_tasks:Running task kernel... 2023-12-07T21:44:18.315 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-07T21:44:18.315 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-07T21:44:18.315 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-07T21:44:18.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T21:44:18.321 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2023-12-07T21:44:18.326 INFO:teuthology.orchestra.run.smithi033.stdout:5.4.0-124-generic 2023-12-07T21:44:18.327 INFO:teuthology.task.kernel:Running kernel on smithi033: 5.4.0-124-generic 2023-12-07T21:44:18.327 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get clean 2023-12-07T21:44:18.432 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get update 2023-12-07T21:44:18.717 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-07T21:44:18.729 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-12-07T21:44:18.779 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-12-07T21:44:19.131 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-12-07T21:44:19.227 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,610 kB] 2023-12-07T21:44:19.372 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,994 kB] 2023-12-07T21:44:19.731 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [686 kB] 2023-12-07T21:44:19.745 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [402 kB] 2023-12-07T21:44:19.754 INFO:teuthology.orchestra.run.smithi033.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-12-07T21:44:19.754 INFO:teuthology.orchestra.run.smithi033.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,411 kB] 2023-12-07T21:44:19.830 INFO:teuthology.orchestra.run.smithi033.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [34.7 kB] 2023-12-07T21:44:19.831 INFO:teuthology.orchestra.run.smithi033.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [336 kB] 2023-12-07T21:44:19.838 INFO:teuthology.orchestra.run.smithi033.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2023-12-07T21:44:19.838 INFO:teuthology.orchestra.run.smithi033.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [634 kB] 2023-12-07T21:44:19.841 INFO:teuthology.orchestra.run.smithi033.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [913 kB] 2023-12-07T21:44:19.852 INFO:teuthology.orchestra.run.smithi033.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [915 kB] 2023-12-07T21:44:19.858 INFO:teuthology.orchestra.run.smithi033.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [484 kB] 2023-12-07T21:44:19.868 INFO:teuthology.orchestra.run.smithi033.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2023-12-07T21:44:19.868 INFO:teuthology.orchestra.run.smithi033.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,528 kB] 2023-12-07T21:44:19.946 INFO:teuthology.orchestra.run.smithi033.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [36.0 kB] 2023-12-07T21:44:19.947 INFO:teuthology.orchestra.run.smithi033.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [353 kB] 2023-12-07T21:44:19.953 INFO:teuthology.orchestra.run.smithi033.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [552 B] 2023-12-07T21:44:20.034 INFO:teuthology.orchestra.run.smithi033.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [192 kB] 2023-12-07T21:44:20.117 INFO:teuthology.orchestra.run.smithi033.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [762 kB] 2023-12-07T21:44:20.465 INFO:teuthology.orchestra.run.smithi033.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2023-12-07T21:44:20.492 INFO:teuthology.orchestra.run.smithi033.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-12-07T21:44:20.492 INFO:teuthology.orchestra.run.smithi033.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,188 B] 2023-12-07T21:44:20.493 INFO:teuthology.orchestra.run.smithi033.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-12-07T21:44:20.493 INFO:teuthology.orchestra.run.smithi033.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-12-07T21:44:20.660 INFO:teuthology.orchestra.run.smithi033.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,141 kB] 2023-12-07T21:44:20.791 INFO:teuthology.orchestra.run.smithi033.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [273 kB] 2023-12-07T21:44:20.813 INFO:teuthology.orchestra.run.smithi033.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2023-12-07T21:44:20.830 INFO:teuthology.orchestra.run.smithi033.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.8 kB] 2023-12-07T21:44:20.848 INFO:teuthology.orchestra.run.smithi033.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2023-12-07T21:44:20.865 INFO:teuthology.orchestra.run.smithi033.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-12-07T21:44:20.883 INFO:teuthology.orchestra.run.smithi033.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-12-07T21:44:20.900 INFO:teuthology.orchestra.run.smithi033.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-12-07T21:44:20.918 INFO:teuthology.orchestra.run.smithi033.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-12-07T21:44:20.936 INFO:teuthology.orchestra.run.smithi033.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-12-07T21:44:20.954 INFO:teuthology.orchestra.run.smithi033.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-12-07T21:44:20.972 INFO:teuthology.orchestra.run.smithi033.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-12-07T21:44:20.989 INFO:teuthology.orchestra.run.smithi033.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-12-07T21:44:23.782 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 18.3 MB in 3s (5,578 kB/s) 2023-12-07T21:44:25.248 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-07T21:44:25.274 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-07T21:44:25.347 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-07T21:44:25.558 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-12-07T21:44:25.559 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-12-07T21:44:25.752 INFO:teuthology.orchestra.run.smithi033.stdout:The following additional packages will be installed: 2023-12-07T21:44:25.752 INFO:teuthology.orchestra.run.smithi033.stdout: linux-generic linux-headers-5.4.0-167 linux-headers-5.4.0-167-generic 2023-12-07T21:44:25.752 INFO:teuthology.orchestra.run.smithi033.stdout: linux-headers-generic linux-image-5.4.0-167-generic 2023-12-07T21:44:25.753 INFO:teuthology.orchestra.run.smithi033.stdout: linux-modules-5.4.0-167-generic linux-modules-extra-5.4.0-167-generic 2023-12-07T21:44:25.754 INFO:teuthology.orchestra.run.smithi033.stdout:Suggested packages: 2023-12-07T21:44:25.754 INFO:teuthology.orchestra.run.smithi033.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-12-07T21:44:25.754 INFO:teuthology.orchestra.run.smithi033.stdout:Recommended packages: 2023-12-07T21:44:25.754 INFO:teuthology.orchestra.run.smithi033.stdout: thermald 2023-12-07T21:44:25.807 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2023-12-07T21:44:25.807 INFO:teuthology.orchestra.run.smithi033.stdout: linux-headers-5.4.0-167 linux-headers-5.4.0-167-generic 2023-12-07T21:44:25.807 INFO:teuthology.orchestra.run.smithi033.stdout: linux-image-5.4.0-167-generic linux-modules-5.4.0-167-generic 2023-12-07T21:44:25.807 INFO:teuthology.orchestra.run.smithi033.stdout: linux-modules-extra-5.4.0-167-generic 2023-12-07T21:44:25.808 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages will be upgraded: 2023-12-07T21:44:25.809 INFO:teuthology.orchestra.run.smithi033.stdout: linux-generic linux-headers-generic linux-image-generic 2023-12-07T21:44:25.873 INFO:teuthology.orchestra.run.smithi033.stdout:3 upgraded, 5 newly installed, 0 to remove and 280 not upgraded. 2023-12-07T21:44:25.873 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 77.0 MB of archives. 2023-12-07T21:44:25.873 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 380 MB of additional disk space will be used. 2023-12-07T21:44:25.873 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-167-generic amd64 5.4.0-167.184 [15.0 MB] 2023-12-07T21:44:26.233 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-167-generic amd64 5.4.0-167.184 [10.5 MB] 2023-12-07T21:44:26.380 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-167-generic amd64 5.4.0-167.184 [39.2 MB] 2023-12-07T21:44:27.117 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.167.164 [1,900 B] 2023-12-07T21:44:27.117 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.167.164 [2,564 B] 2023-12-07T21:44:27.118 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-167 all 5.4.0-167.184 [11.0 MB] 2023-12-07T21:44:27.278 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-167-generic amd64 5.4.0-167.184 [1,367 kB] 2023-12-07T21:44:27.296 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.167.164 [2,428 B] 2023-12-07T21:44:28.003 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 77.0 MB in 1s (52.0 MB/s) 2023-12-07T21:44:28.199 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-modules-5.4.0-167-generic. 2023-12-07T21:44:29.892 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-12-07T21:44:29.897 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../0-linux-modules-5.4.0-167-generic_5.4.0-167.184_amd64.deb ... 2023-12-07T21:44:29.971 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-modules-5.4.0-167-generic (5.4.0-167.184) ... 2023-12-07T21:44:31.930 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-image-5.4.0-167-generic. 2023-12-07T21:44:31.947 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../1-linux-image-5.4.0-167-generic_5.4.0-167.184_amd64.deb ... 2023-12-07T21:44:32.042 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-image-5.4.0-167-generic (5.4.0-167.184) ... 2023-12-07T21:44:33.459 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-167-generic. 2023-12-07T21:44:33.473 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-167-generic_5.4.0-167.184_amd64.deb ... 2023-12-07T21:44:33.517 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-modules-extra-5.4.0-167-generic (5.4.0-167.184) ... 2023-12-07T21:44:38.215 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../3-linux-generic_5.4.0.167.164_amd64.deb ... 2023-12-07T21:44:38.341 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-generic (5.4.0.167.164) over (5.4.0.124.125) ... 2023-12-07T21:44:38.667 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.167.164_amd64.deb ... 2023-12-07T21:44:38.795 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-image-generic (5.4.0.167.164) over (5.4.0.124.125) ... 2023-12-07T21:44:39.123 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-headers-5.4.0-167. 2023-12-07T21:44:39.138 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../5-linux-headers-5.4.0-167_5.4.0-167.184_all.deb ... 2023-12-07T21:44:39.181 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-headers-5.4.0-167 (5.4.0-167.184) ... 2023-12-07T21:44:44.648 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package linux-headers-5.4.0-167-generic. 2023-12-07T21:44:44.669 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../6-linux-headers-5.4.0-167-generic_5.4.0-167.184_amd64.deb ... 2023-12-07T21:44:44.706 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-headers-5.4.0-167-generic (5.4.0-167.184) ... 2023-12-07T21:44:47.431 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.167.164_amd64.deb ... 2023-12-07T21:44:47.577 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking linux-headers-generic (5.4.0.167.164) over (5.4.0.124.125) ... 2023-12-07T21:44:47.919 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-modules-5.4.0-167-generic (5.4.0-167.184) ... 2023-12-07T21:44:49.183 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-headers-5.4.0-167 (5.4.0-167.184) ... 2023-12-07T21:44:49.333 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-headers-5.4.0-167-generic (5.4.0-167.184) ... 2023-12-07T21:44:49.459 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-image-5.4.0-167-generic (5.4.0-167.184) ... 2023-12-07T21:44:50.673 INFO:teuthology.orchestra.run.smithi033.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-167-generic 2023-12-07T21:44:50.673 INFO:teuthology.orchestra.run.smithi033.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-167-generic 2023-12-07T21:44:50.828 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-modules-extra-5.4.0-167-generic (5.4.0-167.184) ... 2023-12-07T21:44:52.054 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-image-generic (5.4.0.167.164) ... 2023-12-07T21:44:52.205 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-headers-generic (5.4.0.167.164) ... 2023-12-07T21:44:52.331 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up linux-generic (5.4.0.167.164) ... 2023-12-07T21:44:52.466 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for linux-image-5.4.0-167-generic (5.4.0-167.184) ... 2023-12-07T21:44:52.543 INFO:teuthology.orchestra.run.smithi033.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-12-07T21:44:52.543 INFO:teuthology.orchestra.run.smithi033.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-167-generic 2023-12-07T21:45:15.528 INFO:teuthology.orchestra.run.smithi033.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-12-07T21:45:15.528 INFO:teuthology.orchestra.run.smithi033.stdout:Sourcing file `/etc/default/grub' 2023-12-07T21:45:15.542 INFO:teuthology.orchestra.run.smithi033.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-07T21:45:15.556 INFO:teuthology.orchestra.run.smithi033.stdout:Generating grub configuration file ... 2023-12-07T21:45:15.841 INFO:teuthology.orchestra.run.smithi033.stdout:Found linux image: /boot/vmlinuz-5.4.0-167-generic 2023-12-07T21:45:15.863 INFO:teuthology.orchestra.run.smithi033.stdout:Found initrd image: /boot/initrd.img-5.4.0-167-generic 2023-12-07T21:45:16.234 INFO:teuthology.orchestra.run.smithi033.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-07T21:45:16.246 INFO:teuthology.orchestra.run.smithi033.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-07T21:45:18.299 INFO:teuthology.orchestra.run.smithi033.stdout:done 2023-12-07T21:45:18.691 DEBUG:teuthology.orchestra.run.smithi033:> dpkg -s linux-image-generic 2023-12-07T21:45:18.739 INFO:teuthology.orchestra.run.smithi033.stdout:Package: linux-image-generic 2023-12-07T21:45:18.739 INFO:teuthology.orchestra.run.smithi033.stdout:Status: install ok installed 2023-12-07T21:45:18.739 INFO:teuthology.orchestra.run.smithi033.stdout:Priority: optional 2023-12-07T21:45:18.739 INFO:teuthology.orchestra.run.smithi033.stdout:Section: kernel 2023-12-07T21:45:18.739 INFO:teuthology.orchestra.run.smithi033.stdout:Installed-Size: 20 2023-12-07T21:45:18.739 INFO:teuthology.orchestra.run.smithi033.stdout:Maintainer: Ubuntu Kernel Team 2023-12-07T21:45:18.739 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: amd64 2023-12-07T21:45:18.739 INFO:teuthology.orchestra.run.smithi033.stdout:Source: linux-meta 2023-12-07T21:45:18.740 INFO:teuthology.orchestra.run.smithi033.stdout:Version: 5.4.0.167.164 2023-12-07T21:45:18.740 INFO:teuthology.orchestra.run.smithi033.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-12-07T21:45:18.740 INFO:teuthology.orchestra.run.smithi033.stdout:Depends: linux-image-5.4.0-167-generic, linux-modules-extra-5.4.0-167-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-07T21:45:18.740 INFO:teuthology.orchestra.run.smithi033.stdout:Recommends: thermald 2023-12-07T21:45:18.740 INFO:teuthology.orchestra.run.smithi033.stdout:Description: Generic Linux kernel image 2023-12-07T21:45:18.740 INFO:teuthology.orchestra.run.smithi033.stdout: This package will always depend on the latest generic kernel image 2023-12-07T21:45:18.740 INFO:teuthology.orchestra.run.smithi033.stdout: available. 2023-12-07T21:45:18.740 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-167-generic 2023-12-07T21:45:18.740 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-07T21:45:18.740 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-07T21:45:18.741 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-07T21:45:18.741 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi033.front.sepia.ceph.com, path=None, version=distro) 2023-12-07T21:45:18.741 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get clean 2023-12-07T21:45:18.840 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get update 2023-12-07T21:45:18.976 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-07T21:45:18.982 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-12-07T21:45:18.999 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-12-07T21:45:18.999 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-12-07T21:45:20.826 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-07T21:45:20.851 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-07T21:45:20.924 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-07T21:45:21.132 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-12-07T21:45:21.132 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-12-07T21:45:21.360 INFO:teuthology.orchestra.run.smithi033.stdout:linux-image-generic is already the newest version (5.4.0.167.164). 2023-12-07T21:45:21.360 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 0 newly installed, 0 to remove and 280 not upgraded. 2023-12-07T21:45:21.362 DEBUG:teuthology.orchestra.run.smithi033:> dpkg -s linux-image-generic 2023-12-07T21:45:21.382 INFO:teuthology.orchestra.run.smithi033.stdout:Package: linux-image-generic 2023-12-07T21:45:21.382 INFO:teuthology.orchestra.run.smithi033.stdout:Status: install ok installed 2023-12-07T21:45:21.382 INFO:teuthology.orchestra.run.smithi033.stdout:Priority: optional 2023-12-07T21:45:21.382 INFO:teuthology.orchestra.run.smithi033.stdout:Section: kernel 2023-12-07T21:45:21.382 INFO:teuthology.orchestra.run.smithi033.stdout:Installed-Size: 20 2023-12-07T21:45:21.382 INFO:teuthology.orchestra.run.smithi033.stdout:Maintainer: Ubuntu Kernel Team 2023-12-07T21:45:21.383 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: amd64 2023-12-07T21:45:21.383 INFO:teuthology.orchestra.run.smithi033.stdout:Source: linux-meta 2023-12-07T21:45:21.383 INFO:teuthology.orchestra.run.smithi033.stdout:Version: 5.4.0.167.164 2023-12-07T21:45:21.383 INFO:teuthology.orchestra.run.smithi033.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-12-07T21:45:21.383 INFO:teuthology.orchestra.run.smithi033.stdout:Depends: linux-image-5.4.0-167-generic, linux-modules-extra-5.4.0-167-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-07T21:45:21.383 INFO:teuthology.orchestra.run.smithi033.stdout:Recommends: thermald 2023-12-07T21:45:21.383 INFO:teuthology.orchestra.run.smithi033.stdout:Description: Generic Linux kernel image 2023-12-07T21:45:21.383 INFO:teuthology.orchestra.run.smithi033.stdout: This package will always depend on the latest generic kernel image 2023-12-07T21:45:21.383 INFO:teuthology.orchestra.run.smithi033.stdout: available. 2023-12-07T21:45:21.383 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2023-12-07T21:45:21.388 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.MX3R6FWcEW 2023-12-07T21:45:21.388 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /boot/grub/grub.cfg /tmp/tmp.MX3R6FWcEW 2023-12-07T21:45:21.399 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.MX3R6FWcEW 2023-12-07T21:45:21.495 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.MX3R6FWcEW is 9KB 2023-12-07T21:45:21.509 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.MX3R6FWcEW 2023-12-07T21:45:21.558 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-12-07T21:45:21.617 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:45:21.617 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-12-07T21:45:21.617 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-12-07T21:45:21.653 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-167-generic 2023-12-07T21:45:21.653 DEBUG:teuthology.orchestra.run.smithi033:> sudo update-grub 2023-12-07T21:45:22.887 INFO:teuthology.orchestra.run.smithi033.stderr:Sourcing file `/etc/default/grub' 2023-12-07T21:45:22.899 INFO:teuthology.orchestra.run.smithi033.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-07T21:45:22.912 INFO:teuthology.orchestra.run.smithi033.stderr:Generating grub configuration file ... 2023-12-07T21:45:23.170 INFO:teuthology.orchestra.run.smithi033.stderr:Found linux image: /boot/vmlinuz-5.4.0-167-generic 2023-12-07T21:45:23.182 INFO:teuthology.orchestra.run.smithi033.stderr:Found initrd image: /boot/initrd.img-5.4.0-167-generic 2023-12-07T21:45:23.545 INFO:teuthology.orchestra.run.smithi033.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-07T21:45:23.557 INFO:teuthology.orchestra.run.smithi033.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-07T21:45:24.629 INFO:teuthology.orchestra.run.smithi033.stderr:done 2023-12-07T21:45:24.631 DEBUG:teuthology.orchestra.run.smithi033:> sudo shutdown -r now 2023-12-07T21:45:54.642 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-07T21:45:54.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-07T21:45:54.644 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T21:46:12.962 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-12-07T21:46:21.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-07T21:46:21.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T21:46:40.354 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-12-07T21:46:52.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-07T21:46:52.362 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T21:46:55.430 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-12-07T21:47:10.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-07T21:47:10.443 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T21:47:10.669 DEBUG:teuthology.orchestra.run.smithi033:> true 2023-12-07T21:47:11.275 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-07T21:47:11.275 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-167-generic"... 2023-12-07T21:47:11.276 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2023-12-07T21:47:11.282 INFO:teuthology.orchestra.run.smithi033.stdout:5.4.0-167-generic 2023-12-07T21:47:11.282 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-167-generic vs 5.4.0-167-generic 2023-12-07T21:47:11.282 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-07T21:47:11.283 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-07T21:47:12.284 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-07T21:47:12.284 DEBUG:teuthology.orchestra.run.smithi033:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-07T21:47:12.379 INFO:teuthology.orchestra.run.smithi033.stdout:ttyS1 2023-12-07T21:47:12.414 DEBUG:teuthology.parallel:result is None 2023-12-07T21:47:12.415 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-07T21:47:12.422 INFO:teuthology.task.internal:Creating test directory... 2023-12-07T21:47:12.422 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-07T21:47:12.429 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-07T21:47:12.445 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-07T21:47:12.477 INFO:teuthology.task.internal:Creating archive directory... 2023-12-07T21:47:12.478 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-07T21:47:12.494 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-07T21:47:12.501 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-07T21:47:12.501 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-07T21:47:12.519 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-07T21:47:12.528 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-07T21:47:12.529 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-07T21:47:12.536 INFO:teuthology.task.internal:Configuring sudo... 2023-12-07T21:47:12.536 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-07T21:47:12.587 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-07T21:47:12.596 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-07T21:47:12.596 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-07T21:47:12.603 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-07T21:47:12.609 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-07T21:47:12.658 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:47:12.659 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-07T21:47:12.692 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2023-12-07T21:47:12.739 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-07T21:47:12.748 INFO:teuthology.task.internal:Starting timer... 2023-12-07T21:47:12.748 INFO:teuthology.run_tasks:Running task pcp... 2023-12-07T21:47:12.763 INFO:teuthology.run_tasks:Running task selinux... 2023-12-07T21:47:12.777 DEBUG:teuthology.task.selinux:Excluding smithi033: OS 'ubuntu' does not support SELinux 2023-12-07T21:47:12.777 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-07T21:47:12.777 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-12-07T21:47:12.777 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-07T21:47:12.778 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-07T21:47:12.815 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-07T21:47:12.816 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-07T21:47:12.832 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-07T21:47:12.833 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi033.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-07T21:53:39.988 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi033.front.sepia.ceph.com')] 2023-12-07T21:53:39.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-07T21:53:39.990 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-07T21:53:40.044 DEBUG:teuthology.orchestra.run.smithi033:> true 2023-12-07T21:53:40.164 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-07T21:53:40.165 INFO:teuthology.run_tasks:Running task clock... 2023-12-07T21:53:40.179 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-07T21:53:40.179 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-07T21:53:40.179 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-07T21:53:40.216 INFO:teuthology.orchestra.run.smithi033.stdout: 7 Dec 21:53:40 ntpd[23537]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-12-07T21:53:40.217 INFO:teuthology.orchestra.run.smithi033.stdout: 7 Dec 21:53:40 ntpd[23537]: Command line: ntpd -gq 2023-12-07T21:53:40.227 INFO:teuthology.orchestra.run.smithi033.stdout: 7 Dec 21:53:40 ntpd[23537]: proto: precision = 0.225 usec (-22) 2023-12-07T21:53:40.227 INFO:teuthology.orchestra.run.smithi033.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-07T21:53:40.227 INFO:teuthology.orchestra.run.smithi033.stdout: 7 Dec 21:53:40 ntpd[23537]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-07T21:53:40.228 INFO:teuthology.orchestra.run.smithi033.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-07T21:53:40.229 INFO:teuthology.orchestra.run.smithi033.stdout: 7 Dec 21:53:40 ntpd[23537]: restrict ::: KOD does nothing without LIMITED. 2023-12-07T21:53:40.229 INFO:teuthology.orchestra.run.smithi033.stdout: 7 Dec 21:53:40 ntpd[23537]: Listen and drop on 0 v6wildcard [::]:123 2023-12-07T21:53:40.229 INFO:teuthology.orchestra.run.smithi033.stdout: 7 Dec 21:53:40 ntpd[23537]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-07T21:53:40.229 INFO:teuthology.orchestra.run.smithi033.stdout: 7 Dec 21:53:40 ntpd[23537]: Listen normally on 2 lo 127.0.0.1:123 2023-12-07T21:53:40.229 INFO:teuthology.orchestra.run.smithi033.stdout: 7 Dec 21:53:40 ntpd[23537]: Listen normally on 3 ens1f0 172.21.15.33:123 2023-12-07T21:53:40.229 INFO:teuthology.orchestra.run.smithi033.stdout: 7 Dec 21:53:40 ntpd[23537]: Listen normally on 4 lo [::1]:123 2023-12-07T21:53:40.229 INFO:teuthology.orchestra.run.smithi033.stdout: 7 Dec 21:53:40 ntpd[23537]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fea5:d294%3]:123 2023-12-07T21:53:40.229 INFO:teuthology.orchestra.run.smithi033.stdout: 7 Dec 21:53:40 ntpd[23537]: Listening on routing socket on fd #22 for interface updates 2023-12-07T21:53:41.217 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:41 ntpd[23537]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-07T21:53:41.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:41 ntpd[23537]: can't open /var/log/ntpstats/rawstats.20231207: Permission denied 2023-12-07T21:53:41.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:41 ntpd[23537]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-07T21:53:41.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:41 ntpd[23537]: can't open /var/log/ntpstats/rawstats.20231207: Permission denied 2023-12-07T21:53:41.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:41 ntpd[23537]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-07T21:53:41.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:41 ntpd[23537]: can't open /var/log/ntpstats/peerstats.20231207: Permission denied 2023-12-07T21:53:42.217 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:42 ntpd[23537]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-07T21:53:42.217 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:42 ntpd[23537]: can't open /var/log/ntpstats/rawstats.20231207: Permission denied 2023-12-07T21:53:43.217 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:43 ntpd[23537]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-07T21:53:43.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:43 ntpd[23537]: can't open /var/log/ntpstats/rawstats.20231207: Permission denied 2023-12-07T21:53:43.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:43 ntpd[23537]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-07T21:53:43.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:43 ntpd[23537]: can't open /var/log/ntpstats/rawstats.20231207: Permission denied 2023-12-07T21:53:43.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:43 ntpd[23537]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-07T21:53:43.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:43 ntpd[23537]: can't open /var/log/ntpstats/peerstats.20231207: Permission denied 2023-12-07T21:53:44.217 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:44 ntpd[23537]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-07T21:53:44.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:44 ntpd[23537]: can't open /var/log/ntpstats/rawstats.20231207: Permission denied 2023-12-07T21:53:44.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:44 ntpd[23537]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-07T21:53:44.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:44 ntpd[23537]: can't open /var/log/ntpstats/rawstats.20231207: Permission denied 2023-12-07T21:53:44.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:44 ntpd[23537]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-07T21:53:44.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:44 ntpd[23537]: can't open /var/log/ntpstats/peerstats.20231207: Permission denied 2023-12-07T21:53:45.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:45 ntpd[23537]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-07T21:53:45.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:45 ntpd[23537]: can't open /var/log/ntpstats/rawstats.20231207: Permission denied 2023-12-07T21:53:45.219 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:45 ntpd[23537]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-07T21:53:45.219 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:45 ntpd[23537]: can't open /var/log/ntpstats/rawstats.20231207: Permission denied 2023-12-07T21:53:45.219 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:45 ntpd[23537]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-07T21:53:45.219 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:45 ntpd[23537]: can't open /var/log/ntpstats/rawstats.20231207: Permission denied 2023-12-07T21:53:45.219 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:45 ntpd[23537]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-07T21:53:45.219 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:45 ntpd[23537]: can't open /var/log/ntpstats/peerstats.20231207: Permission denied 2023-12-07T21:53:45.219 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:45 ntpd[23537]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-07T21:53:45.219 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:45 ntpd[23537]: can't open /var/log/ntpstats/rawstats.20231207: Permission denied 2023-12-07T21:53:45.219 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:45 ntpd[23537]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-07T21:53:45.219 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:45 ntpd[23537]: can't open /var/log/ntpstats/peerstats.20231207: Permission denied 2023-12-07T21:53:46.217 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:46 ntpd[23537]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-07T21:53:46.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:46 ntpd[23537]: can't open /var/log/ntpstats/rawstats.20231207: Permission denied 2023-12-07T21:53:46.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:46 ntpd[23537]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-07T21:53:46.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:46 ntpd[23537]: can't open /var/log/ntpstats/rawstats.20231207: Permission denied 2023-12-07T21:53:46.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:46 ntpd[23537]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-07T21:53:46.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:46 ntpd[23537]: can't open /var/log/ntpstats/peerstats.20231207: Permission denied 2023-12-07T21:53:47.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:47 ntpd[23537]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-07T21:53:47.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:47 ntpd[23537]: can't open /var/log/ntpstats/rawstats.20231207: Permission denied 2023-12-07T21:53:47.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:47 ntpd[23537]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-07T21:53:47.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:47 ntpd[23537]: can't open /var/log/ntpstats/rawstats.20231207: Permission denied 2023-12-07T21:53:47.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:47 ntpd[23537]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-07T21:53:47.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:47 ntpd[23537]: can't open /var/log/ntpstats/rawstats.20231207: Permission denied 2023-12-07T21:53:47.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:47 ntpd[23537]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-07T21:53:47.218 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:47 ntpd[23537]: can't open /var/log/ntpstats/peerstats.20231207: Permission denied 2023-12-07T21:53:47.219 INFO:teuthology.orchestra.run.smithi033.stdout: 7 Dec 21:53:47 ntpd[23537]: ntpd: time slew +0.000798 s 2023-12-07T21:53:47.219 INFO:teuthology.orchestra.run.smithi033.stdout:ntpd: time slew +0.000798s 2023-12-07T21:53:47.219 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:47 ntpd[23537]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-07T21:53:47.219 INFO:teuthology.orchestra.run.smithi033.stderr: 7 Dec 21:53:47 ntpd[23537]: can't open /var/log/ntpstats/loopstats.20231207: Permission denied 2023-12-07T21:53:47.279 INFO:teuthology.orchestra.run.smithi033.stdout: remote refid st t when poll reach delay offset jitter 2023-12-07T21:53:47.279 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================== 2023-12-07T21:53:47.279 INFO:teuthology.orchestra.run.smithi033.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-07T21:53:47.279 INFO:teuthology.orchestra.run.smithi033.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-07T21:53:47.279 INFO:teuthology.orchestra.run.smithi033.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-07T21:53:47.279 INFO:teuthology.orchestra.run.smithi033.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-07T21:53:47.280 INFO:teuthology.run_tasks:Running task install... 2023-12-07T21:53:47.289 DEBUG:teuthology.task.install:project ceph 2023-12-07T21:53:47.289 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'}} 2023-12-07T21:53:47.289 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'} 2023-12-07T21:53:47.290 INFO:teuthology.task.install:Using flavor: default 2023-12-07T21:53:47.296 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-07T21:53:47.297 INFO:teuthology.task.install:extra packages: [] 2023-12-07T21:53:47.298 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-key list | grep Ceph 2023-12-07T21:53:47.417 INFO:teuthology.orchestra.run.smithi033.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-12-07T21:53:47.476 INFO:teuthology.orchestra.run.smithi033.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-07T21:53:47.476 INFO:teuthology.orchestra.run.smithi033.stdout:uid [ unknown] Ceph.com (release key) 2023-12-07T21:53:47.476 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-12-07T21:53:47.477 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-07T21:53:47.705 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default/ 2023-12-07T21:53:47.705 INFO:teuthology.task.install.deb:Package version is 16.2.14-365-g451ceae1-1focal 2023-12-07T21:53:47.820 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:53:47.820 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-07T21:53:47.837 DEBUG:teuthology.orchestra.run.smithi033:> sudo apt-get update 2023-12-07T21:53:48.025 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-07T21:53:48.031 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-12-07T21:53:48.048 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-12-07T21:53:48.141 INFO:teuthology.orchestra.run.smithi033.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-12-07T21:53:48.141 INFO:teuthology.orchestra.run.smithi033.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal InRelease 2023-12-07T21:53:48.184 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal Release [7,680 B] 2023-12-07T21:53:48.222 INFO:teuthology.orchestra.run.smithi033.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal Release.gpg 2023-12-07T21:53:48.259 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 Packages [17.2 kB] 2023-12-07T21:53:48.306 INFO:teuthology.orchestra.run.smithi033.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main i386 Packages [3,106 B] 2023-12-07T21:53:48.691 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 28.0 kB in 1s (43.6 kB/s) 2023-12-07T21:53:49.796 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-07T21:53:49.823 DEBUG:teuthology.orchestra.run.smithi033:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.2.14-365-g451ceae1-1focal cephadm=16.2.14-365-g451ceae1-1focal ceph-mds=16.2.14-365-g451ceae1-1focal ceph-mgr=16.2.14-365-g451ceae1-1focal ceph-common=16.2.14-365-g451ceae1-1focal ceph-fuse=16.2.14-365-g451ceae1-1focal ceph-test=16.2.14-365-g451ceae1-1focal radosgw=16.2.14-365-g451ceae1-1focal python3-rados=16.2.14-365-g451ceae1-1focal python3-rgw=16.2.14-365-g451ceae1-1focal python3-cephfs=16.2.14-365-g451ceae1-1focal python3-rbd=16.2.14-365-g451ceae1-1focal libcephfs2=16.2.14-365-g451ceae1-1focal libcephfs-dev=16.2.14-365-g451ceae1-1focal librados2=16.2.14-365-g451ceae1-1focal librbd1=16.2.14-365-g451ceae1-1focal rbd-fuse=16.2.14-365-g451ceae1-1focal 2023-12-07T21:53:49.899 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-07T21:53:50.108 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-12-07T21:53:50.108 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-12-07T21:53:50.287 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-12-07T21:53:50.288 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-07T21:53:50.288 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-07T21:53:50.288 INFO:teuthology.orchestra.run.smithi033.stdout:The following additional packages will be installed: 2023-12-07T21:53:50.288 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-07T21:53:50.290 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq libjaeger 2023-12-07T21:53:50.290 INFO:teuthology.orchestra.run.smithi033.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2023-12-07T21:53:50.290 INFO:teuthology.orchestra.run.smithi033.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 2023-12-07T21:53:50.290 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph nvme-cli python-pastedeploy-tpl 2023-12-07T21:53:50.290 INFO:teuthology.orchestra.run.smithi033.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2023-12-07T21:53:50.290 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2023-12-07T21:53:50.290 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2023-12-07T21:53:50.290 INFO:teuthology.orchestra.run.smithi033.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-12-07T21:53:50.290 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-12-07T21:53:50.290 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2023-12-07T21:53:50.290 INFO:teuthology.orchestra.run.smithi033.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-12-07T21:53:50.290 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-12-07T21:53:50.290 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-12-07T21:53:50.290 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-12-07T21:53:50.291 INFO:teuthology.orchestra.run.smithi033.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-12-07T21:53:50.291 INFO:teuthology.orchestra.run.smithi033.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-12-07T21:53:50.291 INFO:teuthology.orchestra.run.smithi033.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-12-07T21:53:50.291 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket python3-webtest python3-werkzeug socat xmlstarlet 2023-12-07T21:53:50.292 INFO:teuthology.orchestra.run.smithi033.stdout:Suggested packages: 2023-12-07T21:53:50.292 INFO:teuthology.orchestra.run.smithi033.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2023-12-07T21:53:50.292 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-12-07T21:53:50.292 INFO:teuthology.orchestra.run.smithi033.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-12-07T21:53:50.292 INFO:teuthology.orchestra.run.smithi033.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-12-07T21:53:50.292 INFO:teuthology.orchestra.run.smithi033.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-12-07T21:53:50.292 INFO:teuthology.orchestra.run.smithi033.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-12-07T21:53:50.292 INFO:teuthology.orchestra.run.smithi033.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-07T21:53:50.293 INFO:teuthology.orchestra.run.smithi033.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-12-07T21:53:50.293 INFO:teuthology.orchestra.run.smithi033.stdout:Recommended packages: 2023-12-07T21:53:50.293 INFO:teuthology.orchestra.run.smithi033.stdout: btrfs-tools 2023-12-07T21:53:50.398 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2023-12-07T21:53:50.399 INFO:teuthology.orchestra.run.smithi033.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-07T21:53:50.399 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-07T21:53:50.399 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2023-12-07T21:53:50.400 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2 libjaeger libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2023-12-07T21:53:50.400 INFO:teuthology.orchestra.run.smithi033.stdout: libleveldb1d liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 2023-12-07T21:53:50.400 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph nvme-cli 2023-12-07T21:53:50.401 INFO:teuthology.orchestra.run.smithi033.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-12-07T21:53:50.401 INFO:teuthology.orchestra.run.smithi033.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-12-07T21:53:50.401 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-12-07T21:53:50.401 INFO:teuthology.orchestra.run.smithi033.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-12-07T21:53:50.401 INFO:teuthology.orchestra.run.smithi033.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-12-07T21:53:50.401 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2023-12-07T21:53:50.401 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-12-07T21:53:50.401 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-12-07T21:53:50.401 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-12-07T21:53:50.401 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2023-12-07T21:53:50.401 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2023-12-07T21:53:50.401 INFO:teuthology.orchestra.run.smithi033.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-12-07T21:53:50.401 INFO:teuthology.orchestra.run.smithi033.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2023-12-07T21:53:50.401 INFO:teuthology.orchestra.run.smithi033.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2023-12-07T21:53:50.401 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2023-12-07T21:53:50.402 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages will be upgraded: 2023-12-07T21:53:50.402 INFO:teuthology.orchestra.run.smithi033.stdout: librados2 librbd1 2023-12-07T21:53:50.470 INFO:teuthology.orchestra.run.smithi033.stdout:2 upgraded, 98 newly installed, 0 to remove and 259 not upgraded. 2023-12-07T21:53:50.470 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 174 MB of archives. 2023-12-07T21:53:50.471 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 805 MB of additional disk space will be used. 2023-12-07T21:53:50.471 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-12-07T21:53:50.565 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-12-07T21:53:50.568 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-12-07T21:53:50.577 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-12-07T21:53:50.586 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-12-07T21:53:50.590 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 libjaeger amd64 16.2.14-365-g451ceae1-1focal [3,960 B] 2023-12-07T21:53:50.612 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2023-12-07T21:53:50.619 INFO:teuthology.orchestra.run.smithi033.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-12-07T21:53:50.620 INFO:teuthology.orchestra.run.smithi033.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.2 [211 kB] 2023-12-07T21:53:50.630 INFO:teuthology.orchestra.run.smithi033.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-12-07T21:53:50.630 INFO:teuthology.orchestra.run.smithi033.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-12-07T21:53:50.630 INFO:teuthology.orchestra.run.smithi033.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-12-07T21:53:50.632 INFO:teuthology.orchestra.run.smithi033.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-12-07T21:53:50.633 INFO:teuthology.orchestra.run.smithi033.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-12-07T21:53:50.633 INFO:teuthology.orchestra.run.smithi033.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-12-07T21:53:50.646 INFO:teuthology.orchestra.run.smithi033.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-12-07T21:53:50.647 INFO:teuthology.orchestra.run.smithi033.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-07T21:53:50.648 INFO:teuthology.orchestra.run.smithi033.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-12-07T21:53:50.648 INFO:teuthology.orchestra.run.smithi033.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-12-07T21:53:50.664 INFO:teuthology.orchestra.run.smithi033.stdout:Get:20 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 16.2.14-365-g451ceae1-1focal [3,386 kB] 2023-12-07T21:53:50.665 INFO:teuthology.orchestra.run.smithi033.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-12-07T21:53:50.666 INFO:teuthology.orchestra.run.smithi033.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-12-07T21:53:50.669 INFO:teuthology.orchestra.run.smithi033.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-12-07T21:53:50.671 INFO:teuthology.orchestra.run.smithi033.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-12-07T21:53:50.671 INFO:teuthology.orchestra.run.smithi033.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-12-07T21:53:50.679 INFO:teuthology.orchestra.run.smithi033.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-12-07T21:53:50.685 INFO:teuthology.orchestra.run.smithi033.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-12-07T21:53:50.686 INFO:teuthology.orchestra.run.smithi033.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-12-07T21:53:50.686 INFO:teuthology.orchestra.run.smithi033.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-12-07T21:53:50.687 INFO:teuthology.orchestra.run.smithi033.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-12-07T21:53:50.705 INFO:teuthology.orchestra.run.smithi033.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-12-07T21:53:50.706 INFO:teuthology.orchestra.run.smithi033.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-07T21:53:50.707 INFO:teuthology.orchestra.run.smithi033.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-12-07T21:53:50.709 INFO:teuthology.orchestra.run.smithi033.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-12-07T21:53:50.711 INFO:teuthology.orchestra.run.smithi033.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-12-07T21:53:50.713 INFO:teuthology.orchestra.run.smithi033.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-12-07T21:53:50.727 INFO:teuthology.orchestra.run.smithi033.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-12-07T21:53:50.727 INFO:teuthology.orchestra.run.smithi033.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-12-07T21:53:50.837 INFO:teuthology.orchestra.run.smithi033.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-12-07T21:53:50.860 INFO:teuthology.orchestra.run.smithi033.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-12-07T21:53:50.860 INFO:teuthology.orchestra.run.smithi033.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-12-07T21:53:50.880 INFO:teuthology.orchestra.run.smithi033.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-12-07T21:53:50.880 INFO:teuthology.orchestra.run.smithi033.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-12-07T21:53:50.881 INFO:teuthology.orchestra.run.smithi033.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-12-07T21:53:50.881 INFO:teuthology.orchestra.run.smithi033.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-12-07T21:53:50.881 INFO:teuthology.orchestra.run.smithi033.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-12-07T21:53:50.931 INFO:teuthology.orchestra.run.smithi033.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-12-07T21:53:50.931 INFO:teuthology.orchestra.run.smithi033.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-12-07T21:53:50.931 INFO:teuthology.orchestra.run.smithi033.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-12-07T21:53:50.932 INFO:teuthology.orchestra.run.smithi033.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-12-07T21:53:50.932 INFO:teuthology.orchestra.run.smithi033.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-12-07T21:53:50.936 INFO:teuthology.orchestra.run.smithi033.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-12-07T21:53:50.938 INFO:teuthology.orchestra.run.smithi033.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-12-07T21:53:50.941 INFO:teuthology.orchestra.run.smithi033.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-12-07T21:53:50.949 INFO:teuthology.orchestra.run.smithi033.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-12-07T21:53:50.962 INFO:teuthology.orchestra.run.smithi033.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-12-07T21:53:50.965 INFO:teuthology.orchestra.run.smithi033.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-12-07T21:53:50.969 INFO:teuthology.orchestra.run.smithi033.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-12-07T21:53:50.969 INFO:teuthology.orchestra.run.smithi033.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-12-07T21:53:50.970 INFO:teuthology.orchestra.run.smithi033.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-12-07T21:53:50.970 INFO:teuthology.orchestra.run.smithi033.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-12-07T21:53:50.971 INFO:teuthology.orchestra.run.smithi033.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-12-07T21:53:50.987 INFO:teuthology.orchestra.run.smithi033.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-12-07T21:53:50.987 INFO:teuthology.orchestra.run.smithi033.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-07T21:53:51.001 INFO:teuthology.orchestra.run.smithi033.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-12-07T21:53:51.001 INFO:teuthology.orchestra.run.smithi033.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-12-07T21:53:51.002 INFO:teuthology.orchestra.run.smithi033.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-12-07T21:53:51.002 INFO:teuthology.orchestra.run.smithi033.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-12-07T21:53:51.002 INFO:teuthology.orchestra.run.smithi033.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-12-07T21:53:51.002 INFO:teuthology.orchestra.run.smithi033.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-12-07T21:53:51.005 INFO:teuthology.orchestra.run.smithi033.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-12-07T21:53:51.239 INFO:teuthology.orchestra.run.smithi033.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 16.2.14-365-g451ceae1-1focal [3,388 kB] 2023-12-07T21:53:51.526 INFO:teuthology.orchestra.run.smithi033.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 16.2.14-365-g451ceae1-1focal [729 kB] 2023-12-07T21:53:51.593 INFO:teuthology.orchestra.run.smithi033.stdout:Get:74 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 16.2.14-365-g451ceae1-1focal [414 kB] 2023-12-07T21:53:51.601 INFO:teuthology.orchestra.run.smithi033.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 16.2.14-365-g451ceae1-1focal [22.2 kB] 2023-12-07T21:53:51.602 INFO:teuthology.orchestra.run.smithi033.stdout:Get:76 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 16.2.14-365-g451ceae1-1focal [212 kB] 2023-12-07T21:53:51.629 INFO:teuthology.orchestra.run.smithi033.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 16.2.14-365-g451ceae1-1focal [38.4 kB] 2023-12-07T21:53:51.631 INFO:teuthology.orchestra.run.smithi033.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 16.2.14-365-g451ceae1-1focal [386 kB] 2023-12-07T21:53:51.638 INFO:teuthology.orchestra.run.smithi033.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 16.2.14-365-g451ceae1-1focal [3,706 kB] 2023-12-07T21:53:51.849 INFO:teuthology.orchestra.run.smithi033.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 16.2.14-365-g451ceae1-1focal [116 kB] 2023-12-07T21:53:51.864 INFO:teuthology.orchestra.run.smithi033.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 16.2.14-365-g451ceae1-1focal [448 kB] 2023-12-07T21:53:51.881 INFO:teuthology.orchestra.run.smithi033.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 16.2.14-365-g451ceae1-1focal [22.7 MB] 2023-12-07T21:53:52.734 INFO:teuthology.orchestra.run.smithi033.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 16.2.14-365-g451ceae1-1focal [5,917 kB] 2023-12-07T21:53:52.846 INFO:teuthology.orchestra.run.smithi033.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 16.2.14-365-g451ceae1-1focal [204 kB] 2023-12-07T21:53:52.856 INFO:teuthology.orchestra.run.smithi033.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 16.2.14-365-g451ceae1-1focal [126 kB] 2023-12-07T21:53:52.858 INFO:teuthology.orchestra.run.smithi033.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 16.2.14-365-g451ceae1-1focal [1,551 kB] 2023-12-07T21:53:52.913 INFO:teuthology.orchestra.run.smithi033.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 16.2.14-365-g451ceae1-1focal [6,997 kB] 2023-12-07T21:53:53.144 INFO:teuthology.orchestra.run.smithi033.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 16.2.14-365-g451ceae1-1focal [26.2 MB] 2023-12-07T21:53:54.183 INFO:teuthology.orchestra.run.smithi033.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 16.2.14-365-g451ceae1-1focal [4,048 B] 2023-12-07T21:53:54.184 INFO:teuthology.orchestra.run.smithi033.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 16.2.14-365-g451ceae1-1focal [835 kB] 2023-12-07T21:53:54.195 INFO:teuthology.orchestra.run.smithi033.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 16.2.14-365-g451ceae1-1focal [2,364 kB] 2023-12-07T21:53:54.212 INFO:teuthology.orchestra.run.smithi033.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 16.2.14-365-g451ceae1-1focal [77.4 kB] 2023-12-07T21:53:54.213 INFO:teuthology.orchestra.run.smithi033.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 16.2.14-365-g451ceae1-1focal [100 kB] 2023-12-07T21:53:54.213 INFO:teuthology.orchestra.run.smithi033.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 16.2.14-365-g451ceae1-1focal [2,824 kB] 2023-12-07T21:53:54.280 INFO:teuthology.orchestra.run.smithi033.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 16.2.14-365-g451ceae1-1focal [7,884 kB] 2023-12-07T21:53:54.540 INFO:teuthology.orchestra.run.smithi033.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 16.2.14-365-g451ceae1-1focal [4,164 B] 2023-12-07T21:53:54.541 INFO:teuthology.orchestra.run.smithi033.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 16.2.14-365-g451ceae1-1focal [49.4 MB] 2023-12-07T21:53:56.407 INFO:teuthology.orchestra.run.smithi033.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 16.2.14-365-g451ceae1-1focal [21.5 kB] 2023-12-07T21:53:56.408 INFO:teuthology.orchestra.run.smithi033.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 16.2.14-365-g451ceae1-1focal [11.3 MB] 2023-12-07T21:53:56.675 INFO:teuthology.orchestra.run.smithi033.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2023-12-07-0740-pacific/451ceae11b5d4caee05e94372fe658621eefc0b1/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 16.2.14-365-g451ceae1-1focal [78.6 kB] 2023-12-07T21:53:57.307 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 174 MB in 6s (27.7 MB/s) 2023-12-07T21:53:57.408 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libjaeger. 2023-12-07T21:53:57.467 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141728 files and directories currently installed.) 2023-12-07T21:53:57.471 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../00-libjaeger_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:53:57.515 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libjaeger (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:53:57.843 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-12-07T21:53:57.863 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../01-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-12-07T21:53:57.934 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-12-07T21:53:58.262 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../02-librbd1_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:53:58.372 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librbd1 (16.2.14-365-g451ceae1-1focal) over (15.2.17-0ubuntu0.20.04.5) ... 2023-12-07T21:53:59.096 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../03-librados2_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:53:59.244 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librados2 (16.2.14-365-g451ceae1-1focal) over (15.2.17-0ubuntu0.20.04.5) ... 2023-12-07T21:53:59.925 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libcephfs2. 2023-12-07T21:53:59.941 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../04-libcephfs2_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:53:59.989 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libcephfs2 (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:00.350 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rados. 2023-12-07T21:54:00.366 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../05-python3-rados_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:00.450 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rados (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:00.811 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-07T21:54:00.831 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../06-python3-ceph-argparse_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-07T21:54:00.870 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-ceph-argparse (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:01.139 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cephfs. 2023-12-07T21:54:01.159 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../07-python3-cephfs_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:01.191 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cephfs (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:01.560 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-07T21:54:01.577 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../08-python3-ceph-common_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-07T21:54:01.618 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-ceph-common (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:01.963 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-prettytable. 2023-12-07T21:54:01.983 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../09-python3-prettytable_0.7.2-5_all.deb ... 2023-12-07T21:54:02.021 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-12-07T21:54:02.224 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rbd. 2023-12-07T21:54:02.241 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../10-python3-rbd_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:02.274 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rbd (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:02.627 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-12-07T21:54:02.647 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../11-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-12-07T21:54:02.685 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-12-07T21:54:03.021 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-12-07T21:54:03.040 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../12-librabbitmq4_0.10.0-1_amd64.deb ... 2023-12-07T21:54:03.071 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-12-07T21:54:03.419 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-07T21:54:03.435 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../13-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-12-07T21:54:03.477 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-12-07T21:54:03.830 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package librgw2. 2023-12-07T21:54:03.844 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../14-librgw2_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:03.886 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking librgw2 (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:04.506 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rgw. 2023-12-07T21:54:04.523 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../15-python3-rgw_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:04.555 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rgw (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:04.877 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libleveldb1d:amd64. 2023-12-07T21:54:04.893 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../16-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2023-12-07T21:54:04.933 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2023-12-07T21:54:05.314 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package liboath0. 2023-12-07T21:54:05.329 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../17-liboath0_2.6.1-1.3_amd64.deb ... 2023-12-07T21:54:05.369 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-12-07T21:54:05.647 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libradosstriper1. 2023-12-07T21:54:05.664 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../18-libradosstriper1_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:05.722 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libradosstriper1 (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:06.211 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-common. 2023-12-07T21:54:06.228 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../19-ceph-common_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:06.269 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-common (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:08.491 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-base. 2023-12-07T21:54:08.507 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../20-ceph-base_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:08.560 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-base (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:09.556 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cryptography. 2023-12-07T21:54:09.571 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../21-python3-cryptography_2.8-3ubuntu0.2_amd64.deb ... 2023-12-07T21:54:09.613 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.2) ... 2023-12-07T21:54:10.007 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-openssl. 2023-12-07T21:54:10.027 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../22-python3-openssl_19.0.0-1build1_all.deb ... 2023-12-07T21:54:10.065 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-12-07T21:54:10.352 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-07T21:54:10.371 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../23-ceph-mgr-modules-core_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-07T21:54:10.410 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-modules-core (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:10.806 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-07T21:54:10.826 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../24-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-12-07T21:54:10.867 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-12-07T21:54:11.192 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-07T21:54:11.209 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../25-python3-cherrypy3_8.9.1-6_all.deb ... 2023-12-07T21:54:11.253 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-12-07T21:54:11.612 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jwt. 2023-12-07T21:54:11.629 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../26-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-12-07T21:54:11.672 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-12-07T21:54:12.015 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-logutils. 2023-12-07T21:54:12.033 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../27-python3-logutils_0.3.3-7_all.deb ... 2023-12-07T21:54:12.074 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-12-07T21:54:12.401 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-07T21:54:12.419 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../28-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-12-07T21:54:12.459 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-12-07T21:54:12.762 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-mako. 2023-12-07T21:54:12.778 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../29-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-12-07T21:54:12.828 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-12-07T21:54:13.148 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-07T21:54:13.166 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../30-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-07T21:54:13.240 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-07T21:54:13.551 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-07T21:54:13.569 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../31-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-12-07T21:54:13.643 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-12-07T21:54:13.947 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-webob. 2023-12-07T21:54:13.962 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../32-python3-webob_1%3a1.8.5-2_all.deb ... 2023-12-07T21:54:14.004 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-12-07T21:54:14.356 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-soupsieve. 2023-12-07T21:54:14.373 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../33-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-12-07T21:54:14.415 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-12-07T21:54:14.743 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-bs4. 2023-12-07T21:54:14.762 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../34-python3-bs4_4.8.2-1_all.deb ... 2023-12-07T21:54:14.801 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-12-07T21:54:15.162 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-waitress. 2023-12-07T21:54:15.178 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../35-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-12-07T21:54:15.224 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-12-07T21:54:15.549 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-tempita. 2023-12-07T21:54:15.567 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../36-python3-tempita_0.5.2-6_all.deb ... 2023-12-07T21:54:15.611 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-12-07T21:54:15.935 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-paste. 2023-12-07T21:54:15.956 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../37-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-12-07T21:54:15.993 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-12-07T21:54:16.296 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-07T21:54:16.315 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../38-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-12-07T21:54:16.346 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-12-07T21:54:16.699 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-07T21:54:16.715 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../39-python3-pastedeploy_2.1.0-1_all.deb ... 2023-12-07T21:54:16.757 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-12-07T21:54:17.104 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-webtest. 2023-12-07T21:54:17.118 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../40-python3-webtest_2.0.32-2_all.deb ... 2023-12-07T21:54:17.161 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-12-07T21:54:17.496 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pecan. 2023-12-07T21:54:17.513 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../41-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-12-07T21:54:17.554 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-12-07T21:54:17.926 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-07T21:54:17.941 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../42-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-12-07T21:54:17.984 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-12-07T21:54:18.362 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-07T21:54:18.380 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../43-libsqlite3-mod-ceph_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:18.420 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libsqlite3-mod-ceph (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:18.798 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr. 2023-12-07T21:54:18.819 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../44-ceph-mgr_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:18.848 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:19.360 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mon. 2023-12-07T21:54:19.377 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../45-ceph-mon_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:19.418 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mon (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:20.538 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-osd. 2023-12-07T21:54:20.554 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../46-ceph-osd_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:20.588 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-osd (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:24.193 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph. 2023-12-07T21:54:24.212 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../47-ceph_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:24.251 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:24.595 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-fuse. 2023-12-07T21:54:24.613 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../48-ceph-fuse_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:24.654 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-fuse (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:25.040 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mds. 2023-12-07T21:54:25.060 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../49-ceph-mds_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:25.098 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mds (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:25.660 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package cephadm. 2023-12-07T21:54:25.677 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../50-cephadm_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:25.718 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking cephadm (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:26.063 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-jinja2. 2023-12-07T21:54:26.079 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../51-python3-jinja2_2.10.1-2_all.deb ... 2023-12-07T21:54:26.121 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-12-07T21:54:26.474 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-07T21:54:26.490 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../52-ceph-mgr-cephadm_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-07T21:54:26.532 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-cephadm (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:26.927 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-07T21:54:26.944 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../53-python3-repoze.lru_0.7-2_all.deb ... 2023-12-07T21:54:26.985 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-07T21:54:27.254 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libjs-underscore. 2023-12-07T21:54:27.273 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../54-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-12-07T21:54:27.312 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-12-07T21:54:27.582 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-12-07T21:54:27.600 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../55-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-12-07T21:54:27.673 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-12-07T21:54:27.975 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-routes. 2023-12-07T21:54:27.995 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../56-python3-routes_2.4.1-2_all.deb ... 2023-12-07T21:54:28.057 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-12-07T21:54:28.387 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-07T21:54:28.403 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../57-ceph-mgr-dashboard_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-07T21:54:28.468 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-dashboard (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:29.552 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-decorator. 2023-12-07T21:54:29.572 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../58-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-12-07T21:54:29.602 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-12-07T21:54:29.941 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-12-07T21:54:29.956 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../59-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-12-07T21:54:29.996 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-12-07T21:54:30.332 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-scipy. 2023-12-07T21:54:30.349 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../60-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-12-07T21:54:30.390 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-12-07T21:54:31.766 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-12-07T21:54:31.784 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../61-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-12-07T21:54:31.824 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-12-07T21:54:32.378 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-joblib. 2023-12-07T21:54:32.394 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../62-python3-joblib_0.14.0-3_all.deb ... 2023-12-07T21:54:32.436 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-12-07T21:54:32.839 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-sklearn. 2023-12-07T21:54:32.857 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../63-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-12-07T21:54:32.898 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-12-07T21:54:33.591 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-07T21:54:33.608 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../64-ceph-mgr-diskprediction-local_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-07T21:54:33.649 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-diskprediction-local (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:34.784 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-cachetools. 2023-12-07T21:54:34.803 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../65-python3-cachetools_4.0.0-1_all.deb ... 2023-12-07T21:54:34.841 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-12-07T21:54:35.177 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-07T21:54:35.194 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../66-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-12-07T21:54:35.235 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-12-07T21:54:35.606 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-07T21:54:35.623 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../67-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-12-07T21:54:35.663 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-12-07T21:54:36.013 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-rsa. 2023-12-07T21:54:36.030 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../68-python3-rsa_4.0-3_all.deb ... 2023-12-07T21:54:36.071 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-rsa (4.0-3) ... 2023-12-07T21:54:36.391 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-google-auth. 2023-12-07T21:54:36.411 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../69-python3-google-auth_1.5.1-2_all.deb ... 2023-12-07T21:54:36.449 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-12-07T21:54:36.794 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-blinker. 2023-12-07T21:54:36.815 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../70-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-12-07T21:54:36.853 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-12-07T21:54:37.172 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-oauthlib. 2023-12-07T21:54:37.189 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../71-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-12-07T21:54:37.231 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-12-07T21:54:37.542 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-07T21:54:37.560 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../72-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-12-07T21:54:37.599 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-12-07T21:54:37.936 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-websocket. 2023-12-07T21:54:37.956 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../73-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-12-07T21:54:37.994 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-12-07T21:54:38.322 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-07T21:54:38.340 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../74-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-12-07T21:54:38.380 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-12-07T21:54:38.956 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-07T21:54:38.974 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../75-ceph-mgr-k8sevents_16.2.14-365-g451ceae1-1focal_all.deb ... 2023-12-07T21:54:39.016 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-mgr-k8sevents (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:39.708 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-07T21:54:39.726 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../76-libonig5_6.9.4-1_amd64.deb ... 2023-12-07T21:54:39.766 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-12-07T21:54:40.136 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-07T21:54:40.156 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../77-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-12-07T21:54:40.194 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-12-07T21:54:40.495 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package jq. 2023-12-07T21:54:40.513 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../78-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-12-07T21:54:40.553 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-12-07T21:54:40.839 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package socat. 2023-12-07T21:54:40.860 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../79-socat_1.7.3.3-2_amd64.deb ... 2023-12-07T21:54:40.897 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking socat (1.7.3.3-2) ... 2023-12-07T21:54:41.251 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package xmlstarlet. 2023-12-07T21:54:41.271 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../80-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-12-07T21:54:41.308 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-12-07T21:54:41.757 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package ceph-test. 2023-12-07T21:54:41.774 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../81-ceph-test_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:41.815 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking ceph-test (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:46.213 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-07T21:54:46.232 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../82-libcephfs-dev_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:46.274 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libcephfs-dev (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:46.691 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package nvme-cli. 2023-12-07T21:54:46.707 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../83-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-12-07T21:54:46.791 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-12-07T21:54:47.131 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-atomicwrites. 2023-12-07T21:54:47.151 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../84-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-12-07T21:54:47.222 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-12-07T21:54:47.527 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-attr. 2023-12-07T21:54:47.543 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../85-python3-attr_19.3.0-2_all.deb ... 2023-12-07T21:54:47.602 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-12-07T21:54:47.936 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-webencodings. 2023-12-07T21:54:47.953 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../86-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-12-07T21:54:48.022 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-12-07T21:54:48.339 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-html5lib. 2023-12-07T21:54:48.357 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../87-python3-html5lib_1.0.1-2_all.deb ... 2023-12-07T21:54:48.397 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-12-07T21:54:48.758 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-12-07T21:54:48.774 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../88-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-12-07T21:54:48.816 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-12-07T21:54:49.319 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-packaging. 2023-12-07T21:54:49.336 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../89-python3-packaging_20.3-1_all.deb ... 2023-12-07T21:54:49.377 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-packaging (20.3-1) ... 2023-12-07T21:54:49.714 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pastescript. 2023-12-07T21:54:49.731 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../90-python3-pastescript_2.0.2-4_all.deb ... 2023-12-07T21:54:49.772 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-07T21:54:50.051 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pluggy. 2023-12-07T21:54:50.073 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../91-python3-pluggy_0.13.0-2_all.deb ... 2023-12-07T21:54:50.134 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-12-07T21:54:50.445 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-psutil. 2023-12-07T21:54:50.462 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../92-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-12-07T21:54:50.528 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-12-07T21:54:50.840 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-py. 2023-12-07T21:54:50.860 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../93-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-12-07T21:54:50.898 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-12-07T21:54:51.242 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-07T21:54:51.261 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../94-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-12-07T21:54:51.302 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-12-07T21:54:51.678 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-07T21:54:51.695 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../95-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-12-07T21:54:51.736 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-12-07T21:54:52.074 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-pytest. 2023-12-07T21:54:52.090 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../96-python3-pytest_4.6.9-1_all.deb ... 2023-12-07T21:54:52.131 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-12-07T21:54:52.468 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package python3-simplejson. 2023-12-07T21:54:52.488 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../97-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-12-07T21:54:52.526 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-12-07T21:54:52.982 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package radosgw. 2023-12-07T21:54:52.998 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../98-radosgw_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:53.053 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking radosgw (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:54.232 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package rbd-fuse. 2023-12-07T21:54:54.250 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../99-rbd-fuse_16.2.14-365-g451ceae1-1focal_amd64.deb ... 2023-12-07T21:54:54.299 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking rbd-fuse (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:54.601 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-attr (19.3.0-2) ... 2023-12-07T21:54:54.928 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2023-12-07T21:54:55.062 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-12-07T21:54:55.213 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-12-07T21:54:55.423 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up cephadm (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:56.010 INFO:teuthology.orchestra.run.smithi033.stdout:Adding system user cephadm....done 2023-12-07T21:54:56.171 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-12-07T21:54:56.523 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-07T21:54:56.833 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-12-07T21:54:56.961 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-12-07T21:54:57.293 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-12-07T21:54:57.729 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-12-07T21:54:58.046 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-12-07T21:54:58.348 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-12-07T21:54:58.657 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-12-07T21:54:58.967 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-ceph-argparse (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:54:59.284 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-12-07T21:54:59.403 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-12-07T21:54:59.695 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-12-07T21:55:00.029 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-12-07T21:55:00.464 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-12-07T21:55:00.773 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-12-07T21:55:01.136 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-12-07T21:55:01.437 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libjaeger (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:01.563 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-12-07T21:55:01.906 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-12-07T21:55:02.217 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-packaging (20.3-1) ... 2023-12-07T21:55:02.508 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-12-07T21:55:02.659 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-12-07T21:55:02.960 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-12-07T21:55:03.446 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-12-07T21:55:03.855 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cryptography (2.8-3ubuntu0.2) ... 2023-12-07T21:55:04.265 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up socat (1.7.3.3-2) ... 2023-12-07T21:55:04.391 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-ceph-common (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:04.718 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-12-07T21:55:05.111 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-12-07T21:55:05.246 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-12-07T21:55:05.372 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-12-07T21:55:05.673 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-12-07T21:55:06.002 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-12-07T21:55:06.128 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-12-07T21:55:06.480 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-12-07T21:55:06.781 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-12-07T21:55:07.082 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-12-07T21:55:07.225 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-07T21:55:07.544 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-12-07T21:55:07.852 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-12-07T21:55:08.180 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-12-07T21:55:08.551 INFO:teuthology.orchestra.run.smithi033.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-12-07T21:55:08.858 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-12-07T21:55:08.984 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-12-07T21:55:09.310 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-12-07T21:55:09.686 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-12-07T21:55:09.812 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-12-07T21:55:09.946 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-12-07T21:55:10.447 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-12-07T21:55:11.065 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librados2 (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:11.191 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-12-07T21:55:12.713 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librgw2 (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:12.839 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libsqlite3-mod-ceph (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:12.993 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-12-07T21:55:13.116 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-12-07T21:55:13.516 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-12-07T21:55:13.844 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-12-07T21:55:14.145 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-12-07T21:55:14.547 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libcephfs2 (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:14.698 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-12-07T21:55:15.062 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libradosstriper1 (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:15.187 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-12-07T21:55:15.542 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-12-07T21:55:16.626 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up librbd1 (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:16.888 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-12-07T21:55:17.014 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-12-07T21:55:17.165 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rsa (4.0-3) ... 2023-12-07T21:55:17.483 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-12-07T21:55:17.812 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-modules-core (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:17.935 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-fuse (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:18.200 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-07T21:55:18.200 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-07T21:55:18.656 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libcephfs-dev (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:18.814 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-12-07T21:55:19.124 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-07T21:55:19.584 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-12-07T21:55:19.955 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rados (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:20.078 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rbd (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:20.221 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up rbd-fuse (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:20.339 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-routes (2.4.1-2) ... 2023-12-07T21:55:20.648 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-12-07T21:55:21.035 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-rgw (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:21.162 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-cephfs (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:21.587 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-12-07T21:55:22.524 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-common (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:23.135 INFO:teuthology.orchestra.run.smithi033.stdout:Adding group ceph....done 2023-12-07T21:55:23.554 INFO:teuthology.orchestra.run.smithi033.stdout:Adding system user ceph....done 2023-12-07T21:55:23.732 INFO:teuthology.orchestra.run.smithi033.stdout:Setting system user ceph properties....done 2023-12-07T21:55:23.752 INFO:teuthology.orchestra.run.smithi033.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-07T21:55:23.907 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-07T21:55:24.246 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-07T21:55:24.759 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-test (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:24.886 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up radosgw (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:25.154 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-07T21:55:25.154 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-07T21:55:25.821 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-base (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:26.236 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-07T21:55:26.657 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mds (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:26.882 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-07T21:55:26.882 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-07T21:55:27.333 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:27.567 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-07T21:55:27.567 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-07T21:55:28.016 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-osd (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:28.530 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-07T21:55:28.531 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-07T21:55:28.993 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-k8sevents (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:29.119 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-diskprediction-local (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:29.262 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mon (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:29.523 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-07T21:55:29.523 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-07T21:55:29.946 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-cephadm (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:30.089 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:30.240 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up ceph-mgr-dashboard (16.2.14-365-g451ceae1-1focal) ... 2023-12-07T21:55:30.384 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-12-07T21:55:30.670 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-12-07T21:55:33.614 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-12-07T21:55:34.053 INFO:teuthology.orchestra.run.smithi033.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-07T21:55:34.056 DEBUG:teuthology.parallel:result is None 2023-12-07T21:55:34.056 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=451ceae11b5d4caee05e94372fe658621eefc0b1 2023-12-07T21:55:34.210 DEBUG:teuthology.orchestra.run.smithi033:> dpkg-query -W -f '${Version}' ceph 2023-12-07T21:55:34.232 INFO:teuthology.orchestra.run.smithi033.stdout:16.2.14-365-g451ceae1-1focal 2023-12-07T21:55:34.232 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-365-g451ceae1-1focal 2023-12-07T21:55:34.233 INFO:teuthology.task.install:The correct ceph version 16.2.14-365-g451ceae1-1focal is installed. 2023-12-07T21:55:34.233 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-07T21:55:34.233 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:34.233 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-07T21:55:34.291 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-07T21:55:34.291 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:34.291 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/daemon-helper 2023-12-07T21:55:34.307 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-07T21:55:34.366 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-07T21:55:34.367 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:34.367 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-07T21:55:34.377 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-07T21:55:34.392 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-07T21:55:34.392 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:34.392 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/stdin-killer 2023-12-07T21:55:34.450 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-07T21:55:34.460 INFO:teuthology.run_tasks:Running task ceph... 2023-12-07T21:55:34.557 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-07T21:55:34.557 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /var/log/ceph 2023-12-07T21:55:34.573 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-07T21:55:34.573 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-07T21:55:34.634 INFO:tasks.ceph:Creating extra log directories... 2023-12-07T21:55:34.634 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-07T21:55:34.660 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-07T21:55:34.660 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt kmip ca path': '/etc/ceph/kmiproot.crt', 'rgw crypt kmip client cert': '/etc/ceph/kmip-client.crt', 'rgw crypt kmip client key': '/etc/ceph/kmip-client.key', 'rgw crypt kmip kms key template': 'pykmip-$keyid', 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'kmip', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-07T21:55:34.661 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482401', 'branch': 'wip-yuri5-testing-2023-12-07-0740-pacific', 'description': 'rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7482401', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri5-testing-2023-12-07-0740-pacific'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt kmip ca path': '/etc/ceph/kmiproot.crt', 'rgw crypt kmip client cert': '/etc/ceph/kmip-client.crt', 'rgw crypt kmip client key': '/etc/ceph/kmip-client.key', 'rgw crypt kmip kms key template': 'pykmip-$keyid', 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'kmip', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'}}, 'rgw': {'client.0': {'use-pykmip-role': 'client.0'}}, 'workunit': {'branch': 'wip-yuri5-testing-2023-12-07-0740-pacific', 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 616, 'sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'rgw', 'suite_branch': 'wip-yuri5-testing-2023-12-07-0740-pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '451ceae11b5d4caee05e94372fe658621eefc0b1', 'targets': {'smithi033.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'openssl_keys': {'kmip-client': {'ca': 'kmiproot', 'client': 'client.0', 'cn': 'rgw-client'}, 'kmip-server': {'ca': 'kmiproot', 'client': 'client.0'}, 'kmiproot': {'client': 'client.0', 'cn': 'kmiproot', 'key-type': 'rsa:4096'}}}, {'exec': {'client.0': ['chmod 644 /home/ubuntu/cephtest/ca/kmip-client.key']}}, {'pykmip': {'client.0': {'clientca': 'kmiproot', 'clientcert': 'kmip-client', 'secrets': [{'name': 'pykmip-my-key-1'}, {'name': 'pykmip-my-key-2'}], 'servercert': 'kmip-server'}}}, {'rgw': {'client.0': None}}, {'s3tests': {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-pacific', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-07_21:36:14', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-07T21:55:34.661 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-07T21:55:34.678 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/run/ceph 2023-12-07T21:55:34.734 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:34.734 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2023-12-07T21:55:34.739 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-07T21:55:34.739 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2023-12-07T21:55:34.747 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-07T21:55:34.747 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T21:55:34.747 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 6h/6d Inode: 358 Links: 1 2023-12-07T21:55:34.747 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T21:55:34.749 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-12-07 21:51:54.014696007 +0000 2023-12-07T21:55:34.749 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-12-07 21:51:53.514707509 +0000 2023-12-07T21:55:34.749 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-12-07 21:51:53.514707509 +0000 2023-12-07T21:55:34.749 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2023-12-07T21:55:34.749 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-07T21:55:34.810 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-12-07T21:55:34.810 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-12-07T21:55:34.810 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000427618 s, 1.2 MB/s 2023-12-07T21:55:34.810 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-07T21:55:34.817 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2023-12-07T21:55:34.826 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-07T21:55:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T21:55:34.827 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 6h/6d Inode: 367 Links: 1 2023-12-07T21:55:34.827 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T21:55:34.827 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-12-07 21:51:54.686680545 +0000 2023-12-07T21:55:34.827 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-12-07 21:51:54.242690761 +0000 2023-12-07T21:55:34.827 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-12-07 21:51:54.242690761 +0000 2023-12-07T21:55:34.827 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2023-12-07T21:55:34.827 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-07T21:55:34.889 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-12-07T21:55:34.889 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-12-07T21:55:34.889 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000435855 s, 1.2 MB/s 2023-12-07T21:55:34.890 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-07T21:55:34.899 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2023-12-07T21:55:34.949 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-07T21:55:34.950 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T21:55:34.950 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 6h/6d Inode: 371 Links: 1 2023-12-07T21:55:34.950 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T21:55:34.950 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-12-07 21:51:55.478662325 +0000 2023-12-07T21:55:34.950 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-12-07 21:51:54.926675024 +0000 2023-12-07T21:55:34.950 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-12-07 21:51:54.926675024 +0000 2023-12-07T21:55:34.950 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2023-12-07T21:55:34.950 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-07T21:55:34.963 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-12-07T21:55:34.963 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-12-07T21:55:34.963 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000400305 s, 1.3 MB/s 2023-12-07T21:55:34.964 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-07T21:55:34.973 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2023-12-07T21:55:34.980 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-07T21:55:34.981 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-07T21:55:34.981 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 6h/6d Inode: 377 Links: 1 2023-12-07T21:55:34.981 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-07T21:55:34.981 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-12-07 21:51:56.254644471 +0000 2023-12-07T21:55:34.981 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-12-07 21:51:55.698657263 +0000 2023-12-07T21:55:34.981 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-12-07 21:51:55.698657263 +0000 2023-12-07T21:55:34.981 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: - 2023-12-07T21:55:34.981 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-07T21:55:35.035 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-12-07T21:55:35.035 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-12-07T21:55:35.035 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000409295 s, 1.3 MB/s 2023-12-07T21:55:35.036 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-07T21:55:35.046 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-07T21:55:35.046 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-07T21:55:35.046 INFO:tasks.ceph:Generating config... 2023-12-07T21:55:35.048 INFO:tasks.ceph:[client] debug rgw = 20 2023-12-07T21:55:35.048 INFO:tasks.ceph:[client] rgw crypt kmip ca path = /etc/ceph/kmiproot.crt 2023-12-07T21:55:35.048 INFO:tasks.ceph:[client] rgw crypt kmip client cert = /etc/ceph/kmip-client.crt 2023-12-07T21:55:35.048 INFO:tasks.ceph:[client] rgw crypt kmip client key = /etc/ceph/kmip-client.key 2023-12-07T21:55:35.048 INFO:tasks.ceph:[client] rgw crypt kmip kms key template = pykmip-$keyid 2023-12-07T21:55:35.048 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-12-07T21:55:35.048 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = kmip 2023-12-07T21:55:35.048 INFO:tasks.ceph:[client] setgroup = ceph 2023-12-07T21:55:35.048 INFO:tasks.ceph:[client] setuser = ceph 2023-12-07T21:55:35.048 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-07T21:55:35.048 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-07T21:55:35.048 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-07T21:55:35.048 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-07T21:55:35.048 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-07T21:55:35.049 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-07T21:55:35.049 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-07T21:55:35.049 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-12-07T21:55:35.049 INFO:tasks.ceph:Setting up mon.a... 2023-12-07T21:55:35.049 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-07T21:55:35.098 INFO:teuthology.orchestra.run.smithi033.stdout:creating /etc/ceph/ceph.keyring 2023-12-07T21:55:35.101 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-07T21:55:35.132 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-07T21:55:35.147 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.33')] 2023-12-07T21:55:35.148 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.33'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt kmip ca path': '/etc/ceph/kmiproot.crt', 'rgw crypt kmip client cert': '/etc/ceph/kmip-client.crt', 'rgw crypt kmip client key': '/etc/ceph/kmip-client.key', 'rgw crypt kmip kms key template': 'pykmip-$keyid', 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'kmip', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2023-12-07T21:55:35.148 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:35.148 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-07T21:55:35.153 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.33 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-07T21:55:35.185 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setuser ceph since I am not root 2023-12-07T21:55:35.185 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setgroup ceph since I am not root 2023-12-07T21:55:35.185 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-07T21:55:35.186 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: generated fsid d80f57a9-566f-4802-89c6-ac2d96473541 2023-12-07T21:55:35.186 INFO:teuthology.orchestra.run.smithi033.stdout:epoch 0 2023-12-07T21:55:35.186 INFO:teuthology.orchestra.run.smithi033.stdout:fsid d80f57a9-566f-4802-89c6-ac2d96473541 2023-12-07T21:55:35.186 INFO:teuthology.orchestra.run.smithi033.stdout:last_changed 2023-12-07T21:55:35.184794+0000 2023-12-07T21:55:35.186 INFO:teuthology.orchestra.run.smithi033.stdout:created 2023-12-07T21:55:35.184794+0000 2023-12-07T21:55:35.186 INFO:teuthology.orchestra.run.smithi033.stdout:min_mon_release 0 (unknown) 2023-12-07T21:55:35.186 INFO:teuthology.orchestra.run.smithi033.stdout:election_strategy: 1 2023-12-07T21:55:35.186 INFO:teuthology.orchestra.run.smithi033.stdout:0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2023-12-07T21:55:35.186 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-12-07T21:55:35.187 DEBUG:teuthology.orchestra.run.smithi033:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-07T21:55:35.193 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID d80f57a9-566f-4802-89c6-ac2d96473541... 2023-12-07T21:55:35.194 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-07T21:55:35.228 INFO:teuthology.orchestra.run.smithi033.stdout:[global] 2023-12-07T21:55:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: chdir = "" 2023-12-07T21:55:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-07T21:55:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: auth supported = cephx 2023-12-07T21:55:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: filestore xattr use omap = true 2023-12-07T21:55:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: mon clock drift allowed = 1.000 2023-12-07T21:55:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: osd crush chooseleaf type = 0 2023-12-07T21:55:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: auth debug = true 2023-12-07T21:55:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on old message = true 2023-12-07T21:55:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on bug = true 2023-12-07T21:55:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-07T21:55:35.230 INFO:teuthology.orchestra.run.smithi033.stdout: mon pg warn max object skew = 0 2023-12-07T21:55:35.230 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.230 INFO:teuthology.orchestra.run.smithi033.stdout: # disable pg_autoscaler by default for new pools 2023-12-07T21:55:35.237 INFO:teuthology.orchestra.run.smithi033.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-07T21:55:35.237 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.237 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default size = 2 2023-12-07T21:55:35.237 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.237 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow primary affinity = true 2023-12-07T21:55:35.238 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow pg remap = true 2023-12-07T21:55:35.238 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on legacy crush tunables = false 2023-12-07T21:55:35.238 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on crush straw calc version zero = false 2023-12-07T21:55:35.238 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on no sortbitwise = false 2023-12-07T21:55:35.238 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on osd down out interval zero = false 2023-12-07T21:55:35.238 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on too few osds = false 2023-12-07T21:55:35.238 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-07T21:55:35.238 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-07T21:55:35.238 INFO:teuthology.orchestra.run.smithi033.stdout: mon_allow_pool_size_one = true 2023-12-07T21:55:35.238 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.238 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-12-07T21:55:35.238 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.238 INFO:teuthology.orchestra.run.smithi033.stdout: osd default data pool replay window = 5 2023-12-07T21:55:35.238 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.238 INFO:teuthology.orchestra.run.smithi033.stdout: mon allow pool delete = true 2023-12-07T21:55:35.239 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.239 INFO:teuthology.orchestra.run.smithi033.stdout: mon cluster log file level = debug 2023-12-07T21:55:35.239 INFO:teuthology.orchestra.run.smithi033.stdout: debug asserts on shutdown = true 2023-12-07T21:55:35.239 INFO:teuthology.orchestra.run.smithi033.stdout: mon health detail to clog = false 2023-12-07T21:55:35.239 INFO:teuthology.orchestra.run.smithi033.stdout: mon host = 172.21.15.33 2023-12-07T21:55:35.239 INFO:teuthology.orchestra.run.smithi033.stdout: fsid = d80f57a9-566f-4802-89c6-ac2d96473541 2023-12-07T21:55:35.239 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.239 INFO:teuthology.orchestra.run.smithi033.stdout:[osd] 2023-12-07T21:55:35.239 INFO:teuthology.orchestra.run.smithi033.stdout: osd journal size = 100 2023-12-07T21:55:35.239 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.239 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub load threshold = 5.0 2023-12-07T21:55:35.239 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub max interval = 600 2023-12-07T21:55:35.239 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.239 INFO:teuthology.orchestra.run.smithi033.stdout: osd recover clone overlap = true 2023-12-07T21:55:35.239 INFO:teuthology.orchestra.run.smithi033.stdout: osd recovery max chunk = 1048576 2023-12-07T21:55:35.239 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.240 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug shutdown = true 2023-12-07T21:55:35.240 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug op order = true 2023-12-07T21:55:35.240 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify stray on activate = true 2023-12-07T21:55:35.240 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.240 INFO:teuthology.orchestra.run.smithi033.stdout: osd open classes on start = true 2023-12-07T21:55:35.240 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug pg log writeout = true 2023-12-07T21:55:35.240 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.240 INFO:teuthology.orchestra.run.smithi033.stdout: osd deep scrub update digest min age = 30 2023-12-07T21:55:35.240 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.240 INFO:teuthology.orchestra.run.smithi033.stdout: osd map max advance = 10 2023-12-07T21:55:35.240 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.241 INFO:teuthology.orchestra.run.smithi033.stdout: journal zero on create = true 2023-12-07T21:55:35.241 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.241 INFO:teuthology.orchestra.run.smithi033.stdout: filestore ondisk finisher threads = 3 2023-12-07T21:55:35.241 INFO:teuthology.orchestra.run.smithi033.stdout: filestore apply finisher threads = 3 2023-12-07T21:55:35.241 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.241 INFO:teuthology.orchestra.run.smithi033.stdout: bdev debug aio = true 2023-12-07T21:55:35.241 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug misdirected ops = true 2023-12-07T21:55:35.241 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-12-07T21:55:35.241 INFO:teuthology.orchestra.run.smithi033.stdout: debug osd = 20 2023-12-07T21:55:35.241 INFO:teuthology.orchestra.run.smithi033.stdout: osd shutdown pgref assert = True 2023-12-07T21:55:35.241 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.241 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr] 2023-12-07T21:55:35.241 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-12-07T21:55:35.241 INFO:teuthology.orchestra.run.smithi033.stdout: debug mgr = 20 2023-12-07T21:55:35.242 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2023-12-07T21:55:35.242 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2023-12-07T21:55:35.242 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2023-12-07T21:55:35.242 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2023-12-07T21:55:35.242 INFO:teuthology.orchestra.run.smithi033.stdout: mgr/telemetry/nag = false 2023-12-07T21:55:35.242 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.242 INFO:teuthology.orchestra.run.smithi033.stdout:[mon] 2023-12-07T21:55:35.242 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-12-07T21:55:35.242 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2023-12-07T21:55:35.242 INFO:teuthology.orchestra.run.smithi033.stdout: debug paxos = 20 2023-12-07T21:55:35.242 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2023-12-07T21:55:35.242 INFO:teuthology.orchestra.run.smithi033.stdout: mon data avail warn = 5 2023-12-07T21:55:35.242 INFO:teuthology.orchestra.run.smithi033.stdout: mon mgr mkfs grace = 240 2023-12-07T21:55:35.242 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2023-12-07T21:55:35.242 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd reporter subtree level = osd 2023-12-07T21:55:35.243 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd prime pg temp = true 2023-12-07T21:55:35.243 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2023-12-07T21:55:35.243 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.243 INFO:teuthology.orchestra.run.smithi033.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-07T21:55:35.243 INFO:teuthology.orchestra.run.smithi033.stdout: auth mon ticket ttl = 660 # 11m 2023-12-07T21:55:35.243 INFO:teuthology.orchestra.run.smithi033.stdout: auth service ticket ttl = 240 # 4m 2023-12-07T21:55:35.243 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.243 INFO:teuthology.orchestra.run.smithi033.stdout: # don't complain about insecure global_id in the test suite 2023-12-07T21:55:35.243 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-07T21:55:35.243 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-07T21:55:35.243 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:35.243 INFO:teuthology.orchestra.run.smithi033.stdout:[client] 2023-12-07T21:55:35.243 INFO:teuthology.orchestra.run.smithi033.stdout: rgw cache enabled = true 2023-12-07T21:55:35.243 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable ops log = true 2023-12-07T21:55:35.243 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable usage log = true 2023-12-07T21:55:35.244 INFO:teuthology.orchestra.run.smithi033.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-07T21:55:35.244 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-07T21:55:35.244 INFO:teuthology.orchestra.run.smithi033.stdout: debug rgw = 20 2023-12-07T21:55:35.244 INFO:teuthology.orchestra.run.smithi033.stdout: rgw crypt kmip ca path = /etc/ceph/kmiproot.crt 2023-12-07T21:55:35.244 INFO:teuthology.orchestra.run.smithi033.stdout: rgw crypt kmip client cert = /etc/ceph/kmip-client.crt 2023-12-07T21:55:35.244 INFO:teuthology.orchestra.run.smithi033.stdout: rgw crypt kmip client key = /etc/ceph/kmip-client.key 2023-12-07T21:55:35.244 INFO:teuthology.orchestra.run.smithi033.stdout: rgw crypt kmip kms key template = pykmip-$keyid 2023-12-07T21:55:35.244 INFO:teuthology.orchestra.run.smithi033.stdout: rgw crypt require ssl = False 2023-12-07T21:55:35.244 INFO:teuthology.orchestra.run.smithi033.stdout: rgw crypt s3 kms backend = kmip 2023-12-07T21:55:35.244 INFO:teuthology.orchestra.run.smithi033.stdout: setgroup = ceph 2023-12-07T21:55:35.244 INFO:teuthology.orchestra.run.smithi033.stdout: setuser = ceph 2023-12-07T21:55:35.244 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.a] 2023-12-07T21:55:35.245 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-07T21:55:35.245 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-07T21:55:35.334 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-07T21:55:35.334 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:35.334 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-07T21:55:35.341 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:35.341 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-07T21:55:35.346 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi033.front.sepia.ceph.com 2023-12-07T21:55:35.346 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:35.347 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-07T21:55:35.347 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-07T21:55:35.371 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:35.371 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-07T21:55:35.420 INFO:tasks.ceph:Setting up mon nodes... 2023-12-07T21:55:35.421 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-07T21:55:35.421 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-07T21:55:35.466 INFO:teuthology.orchestra.run.smithi033.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-07T21:55:35.467 INFO:tasks.ceph:Setting up mds nodes... 2023-12-07T21:55:35.467 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-07T21:55:35.467 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-07T21:55:35.498 INFO:teuthology.orchestra.run.smithi033.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-07T21:55:35.512 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-07T21:55:35.512 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-07T21:55:35.513 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-07T21:55:35.569 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-07T21:55:35.569 INFO:tasks.ceph:role: osd.0 2023-12-07T21:55:35.569 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-07T21:55:35.569 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-07T21:55:36.107 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T21:55:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T21:55:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T21:55:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 2023-12-07T21:55:36.107 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T21:55:36.107 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-12-07T21:55:36.108 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T21:55:36.108 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-07T21:55:36.108 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T21:55:36.108 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T21:55:36.110 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-12-07T21:55:36.110 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-07T21:55:36.198 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-07T21:55:36.210 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2023-12-07T21:55:36.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T21:55:36.212 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-07T21:55:36.223 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-07T21:55:36.223 INFO:tasks.ceph:role: osd.1 2023-12-07T21:55:36.223 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-07T21:55:36.224 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-07T21:55:36.743 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T21:55:36.743 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T21:55:36.743 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T21:55:36.743 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 2023-12-07T21:55:36.743 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T21:55:36.743 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-12-07T21:55:36.744 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T21:55:36.744 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-07T21:55:36.744 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T21:55:36.744 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T21:55:36.752 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-12-07T21:55:36.752 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-07T21:55:36.785 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-07T21:55:36.838 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2023-12-07T21:55:36.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T21:55:36.839 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-07T21:55:36.850 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-07T21:55:36.850 INFO:tasks.ceph:role: osd.2 2023-12-07T21:55:36.850 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-07T21:55:36.850 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-07T21:55:37.339 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-07T21:55:37.339 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-07T21:55:37.339 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-07T21:55:37.340 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 2023-12-07T21:55:37.340 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-07T21:55:37.340 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-12-07T21:55:37.340 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-07T21:55:37.340 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-07T21:55:37.340 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-07T21:55:37.340 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-07T21:55:37.341 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-12-07T21:55:37.341 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-07T21:55:37.393 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-07T21:55:37.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T21:55:37.405 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: /sbin/restorecon: command not found 2023-12-07T21:55:37.406 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T21:55:37.486 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:37.478+0000 7f08f9731200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-07T21:55:37.488 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:37.482+0000 7f08f9731200 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-07T21:55:37.488 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:37.482+0000 7f08f9731200 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-07T21:55:37.488 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:37.482+0000 7f08f9731200 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-07T21:55:37.488 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:37.482+0000 7f08f9731200 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-07T21:55:37.493 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:37.486+0000 7f08f9731200 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-07T21:55:39.674 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-07T21:55:39.693 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T21:55:39.729 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:39.722+0000 7f6981ce7200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-07T21:55:39.729 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:39.722+0000 7f6981ce7200 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-07T21:55:39.729 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:39.722+0000 7f6981ce7200 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-07T21:55:39.729 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:39.722+0000 7f6981ce7200 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-07T21:55:39.729 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:39.722+0000 7f6981ce7200 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-07T21:55:39.730 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:39.722+0000 7f6981ce7200 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-07T21:55:41.935 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-07T21:55:41.951 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-07T21:55:41.989 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:41.982+0000 7f5fb0899200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-07T21:55:41.989 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:41.982+0000 7f5fb0899200 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-07T21:55:41.989 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:41.982+0000 7f5fb0899200 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-07T21:55:41.989 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:41.982+0000 7f5fb0899200 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-07T21:55:41.989 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:41.982+0000 7f5fb0899200 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-07T21:55:41.989 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:55:41.982+0000 7f5fb0899200 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-07T21:55:44.181 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-07T21:55:44.198 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-07T21:55:44.199 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:44.199 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-07T21:55:44.256 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:44.256 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-07T21:55:44.268 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:44.268 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-07T21:55:44.279 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:44.279 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-07T21:55:44.293 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:44.293 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-07T21:55:44.299 INFO:tasks.ceph:Adding keys to all mons... 2023-12-07T21:55:44.299 DEBUG:teuthology.orchestra.run.smithi033:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-07T21:55:44.313 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr.x] 2023-12-07T21:55:44.313 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBXP3JllOmnGxAATunm0IX611mJW03QWgLZjg== 2023-12-07T21:55:44.313 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.0] 2023-12-07T21:55:44.313 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBZP3JloebwHBAAr3vT4JCny0yzqxtfWKpCPg== 2023-12-07T21:55:44.313 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.1] 2023-12-07T21:55:44.313 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBbP3JlICFmKxAABvs0Wi2AudzuV+BPcFODrw== 2023-12-07T21:55:44.313 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.2] 2023-12-07T21:55:44.313 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBdP3JlTCHcOhAAEAWl07NyKp73Di5CzIKZXw== 2023-12-07T21:55:44.313 INFO:teuthology.orchestra.run.smithi033.stdout:[client.0] 2023-12-07T21:55:44.313 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBXP3JlFCGxHRAAJEuKKkSn+VZkMcy1TP0rgA== 2023-12-07T21:55:44.314 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-07T21:55:44.400 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T21:55:44.443 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T21:55:44.486 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-07T21:55:44.575 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-07T21:55:44.624 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-07T21:55:44.624 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-07T21:55:44.638 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-07T21:55:44.739 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-07T21:55:44.755 DEBUG:teuthology.orchestra.run.smithi033:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-07T21:55:44.759 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-07T21:55:44.760 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-07T21:55:44.760 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-07T21:55:44.804 INFO:tasks.ceph.mon.a:Started 2023-12-07T21:55:44.804 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-07T21:55:44.804 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-07T21:55:44.804 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-07T21:55:44.808 INFO:tasks.ceph.mgr.x:Started 2023-12-07T21:55:44.808 INFO:tasks.ceph:Setting crush tunables to default 2023-12-07T21:55:44.808 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd crush tunables default 2023-12-07T21:55:45.021 INFO:teuthology.orchestra.run.smithi033.stderr:adjusted tunables profile to default 2023-12-07T21:55:45.032 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-07T21:55:45.032 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:45.032 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-07T21:55:45.042 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:45.042 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-07T21:55:45.101 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:55:45.101 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-07T21:55:45.111 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new dc992a6a-05d7-42e3-b943-34ea323f923c 0 2023-12-07T21:55:45.141 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:45.134+0000 7ff4b07ac500 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-07T21:55:45.202 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:45.194+0000 7ff4b07ac500 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-07T21:55:45.309 INFO:teuthology.orchestra.run.smithi033.stdout:0 2023-12-07T21:55:45.327 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new 7c55af6b-1aeb-49a6-94bd-3faccf2d6445 1 2023-12-07T21:55:45.574 INFO:teuthology.orchestra.run.smithi033.stdout:1 2023-12-07T21:55:45.587 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new 5c54b971-af63-4b5e-898a-ee8072a7524b 2 2023-12-07T21:55:45.777 INFO:teuthology.orchestra.run.smithi033.stdout:2 2023-12-07T21:55:45.788 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-07T21:55:45.789 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-12-07T21:55:45.791 INFO:tasks.ceph.osd.0:Started 2023-12-07T21:55:45.791 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-07T21:55:45.791 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-12-07T21:55:45.793 INFO:tasks.ceph.osd.1:Started 2023-12-07T21:55:45.794 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-07T21:55:45.794 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-12-07T21:55:45.796 INFO:tasks.ceph.osd.2:Started 2023-12-07T21:55:45.796 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-07T21:55:45.864 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:45.854+0000 7ff4b07ac500 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-07T21:55:45.974 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:45.975 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":5,"fsid":"d80f57a9-566f-4802-89c6-ac2d96473541","created":"2023-12-07T21:55:44.914448+0000","modified":"2023-12-07T21:55:45.771064+0000","last_up_change":"0.000000","last_in_change":"2023-12-07T21:55:45.771064+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"dc992a6a-05d7-42e3-b943-34ea323f923c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7c55af6b-1aeb-49a6-94bd-3faccf2d6445","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5c54b971-af63-4b5e-898a-ee8072a7524b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T21:55:45.986 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-07T21:55:45.986 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-07T21:55:46.156 INFO:tasks.ceph.osd.1.smithi033.stderr:2023-12-07T21:55:46.150+0000 7f7df614f200 -1 Falling back to public interface 2023-12-07T21:55:46.172 INFO:tasks.ceph.osd.2.smithi033.stderr:2023-12-07T21:55:46.166+0000 7f1b3ef60200 -1 Falling back to public interface 2023-12-07T21:55:46.188 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-12-07T21:55:46.182+0000 7f032ad0e200 -1 Falling back to public interface 2023-12-07T21:55:46.288 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-07T21:55:46.444 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:46.438+0000 7ff4b07ac500 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-07T21:55:46.453 INFO:teuthology.misc.health.smithi033.stdout: 2023-12-07T21:55:46.453 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":5,"fsid":"d80f57a9-566f-4802-89c6-ac2d96473541","created":"2023-12-07T21:55:44.914448+0000","modified":"2023-12-07T21:55:45.771064+0000","last_up_change":"0.000000","last_in_change":"2023-12-07T21:55:45.771064+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"dc992a6a-05d7-42e3-b943-34ea323f923c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7c55af6b-1aeb-49a6-94bd-3faccf2d6445","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5c54b971-af63-4b5e-898a-ee8072a7524b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T21:55:46.464 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-07T21:55:46.601 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:46.594+0000 7ff4b07ac500 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-07T21:55:46.832 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:46.826+0000 7ff4b07ac500 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-07T21:55:46.992 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:46.986+0000 7ff4b07ac500 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-07T21:55:47.045 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:47.038+0000 7ff4b07ac500 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-07T21:55:47.109 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:47.102+0000 7ff4b07ac500 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-07T21:55:47.166 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:47.158+0000 7ff4b07ac500 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-07T21:55:47.464 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:47.458+0000 7ff4b07ac500 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-07T21:55:47.613 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:47.606+0000 7ff4b07ac500 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-07T21:55:47.864 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:47.858+0000 7ff4b07ac500 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-07T21:55:47.930 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:47.922+0000 7ff4b07ac500 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-07T21:55:48.160 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:48.154+0000 7ff4b07ac500 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-07T21:55:48.219 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:48.214+0000 7ff4b07ac500 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-07T21:55:48.369 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:48.362+0000 7ff4b07ac500 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-07T21:55:48.469 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:48.458+0000 7ff4b07ac500 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-07T21:55:48.610 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:48.602+0000 7ff4b07ac500 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-07T21:55:48.806 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:48.798+0000 7ff4b07ac500 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-07T21:55:48.954 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-12-07T21:55:48.946+0000 7f032ad0e200 -1 osd.0 0 log_to_monitors {default=true} 2023-12-07T21:55:48.958 INFO:tasks.ceph.osd.1.smithi033.stderr:2023-12-07T21:55:48.954+0000 7f7df614f200 -1 osd.1 0 log_to_monitors {default=true} 2023-12-07T21:55:49.189 INFO:tasks.ceph.osd.2.smithi033.stderr:2023-12-07T21:55:49.182+0000 7f1b3ef60200 -1 osd.2 0 log_to_monitors {default=true} 2023-12-07T21:55:49.842 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-07T21:55:49.834+0000 7ff49a48c700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-07T21:55:50.913 INFO:tasks.ceph.osd.1.smithi033.stderr:2023-12-07T21:55:50.906+0000 7f7df18ef700 -1 osd.1 0 waiting for initial osdmap 2023-12-07T21:55:50.913 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-12-07T21:55:50.906+0000 7f03264ae700 -1 osd.0 0 waiting for initial osdmap 2023-12-07T21:55:50.914 INFO:tasks.ceph.osd.2.smithi033.stderr:2023-12-07T21:55:50.910+0000 7f1b3a700700 -1 osd.2 0 waiting for initial osdmap 2023-12-07T21:55:50.917 INFO:tasks.ceph.osd.1.smithi033.stderr:2023-12-07T21:55:50.910+0000 7f7dee663700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T21:55:50.918 INFO:tasks.ceph.osd.0.smithi033.stderr:2023-12-07T21:55:50.914+0000 7f0323222700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T21:55:50.919 INFO:tasks.ceph.osd.2.smithi033.stderr:2023-12-07T21:55:50.914+0000 7f1b37474700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-07T21:55:52.768 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-07T21:55:53.055 INFO:teuthology.misc.health.smithi033.stdout: 2023-12-07T21:55:53.055 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":9,"fsid":"d80f57a9-566f-4802-89c6-ac2d96473541","created":"2023-12-07T21:55:44.914448+0000","modified":"2023-12-07T21:55:52.904771+0000","last_up_change":"2023-12-07T21:55:51.902623+0000","last_in_change":"2023-12-07T21:55:45.771064+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-07T21:55:51.973567+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"dc992a6a-05d7-42e3-b943-34ea323f923c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":29685},{"type":"v1","addr":"172.21.15.33:6817","nonce":29685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":29685},{"type":"v1","addr":"172.21.15.33:6819","nonce":29685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":29685},{"type":"v1","addr":"172.21.15.33:6823","nonce":29685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":29685},{"type":"v1","addr":"172.21.15.33:6821","nonce":29685}]},"public_addr":"172.21.15.33:6817/29685","cluster_addr":"172.21.15.33:6819/29685","heartbeat_back_addr":"172.21.15.33:6823/29685","heartbeat_front_addr":"172.21.15.33:6821/29685","state":["exists","up"]},{"osd":1,"uuid":"7c55af6b-1aeb-49a6-94bd-3faccf2d6445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":29683},{"type":"v1","addr":"172.21.15.33:6801","nonce":29683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":29683},{"type":"v1","addr":"172.21.15.33:6803","nonce":29683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":29683},{"type":"v1","addr":"172.21.15.33:6807","nonce":29683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":29683},{"type":"v1","addr":"172.21.15.33:6805","nonce":29683}]},"public_addr":"172.21.15.33:6801/29683","cluster_addr":"172.21.15.33:6803/29683","heartbeat_back_addr":"172.21.15.33:6807/29683","heartbeat_front_addr":"172.21.15.33:6805/29683","state":["exists","up"]},{"osd":2,"uuid":"5c54b971-af63-4b5e-898a-ee8072a7524b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":29684},{"type":"v1","addr":"172.21.15.33:6809","nonce":29684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":29684},{"type":"v1","addr":"172.21.15.33:6811","nonce":29684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":29684},{"type":"v1","addr":"172.21.15.33:6815","nonce":29684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":29684},{"type":"v1","addr":"172.21.15.33:6813","nonce":29684}]},"public_addr":"172.21.15.33:6809/29684","cluster_addr":"172.21.15.33:6811/29684","heartbeat_back_addr":"172.21.15.33:6815/29684","heartbeat_front_addr":"172.21.15.33:6813/29684","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T21:55:49.936627+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T21:55:49.941640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T21:55:50.195038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T21:55:53.068 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-07T21:55:53.069 INFO:tasks.ceph:Creating RBD pool 2023-12-07T21:55:53.069 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-07T21:55:53.930 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'rbd' created 2023-12-07T21:55:53.941 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-07T21:55:54.946 INFO:teuthology.orchestra.run.smithi033.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-07T21:55:54.958 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-07T21:55:54.958 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-07T21:55:54.958 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-07T21:55:54.958 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-07T21:55:54.959 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-07T21:55:55.279 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:55.293 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":29496},{"type":"v1","addr":"172.21.15.33:6825","nonce":29496}]},"active_addr":"172.21.15.33:6825/29496","active_change":"2023-12-07T21:55:48.827509+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":3900296319}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":3906650020}]}]} 2023-12-07T21:55:55.295 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-07T21:55:55.295 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-07T21:55:55.295 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-07T21:55:55.574 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:55.574 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":11,"fsid":"d80f57a9-566f-4802-89c6-ac2d96473541","created":"2023-12-07T21:55:44.914448+0000","modified":"2023-12-07T21:55:54.917517+0000","last_up_change":"2023-12-07T21:55:51.902623+0000","last_in_change":"2023-12-07T21:55:45.771064+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-07T21:55:51.973567+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-07T21:55:53.407039+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"dc992a6a-05d7-42e3-b943-34ea323f923c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":29685},{"type":"v1","addr":"172.21.15.33:6817","nonce":29685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":29685},{"type":"v1","addr":"172.21.15.33:6819","nonce":29685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":29685},{"type":"v1","addr":"172.21.15.33:6823","nonce":29685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":29685},{"type":"v1","addr":"172.21.15.33:6821","nonce":29685}]},"public_addr":"172.21.15.33:6817/29685","cluster_addr":"172.21.15.33:6819/29685","heartbeat_back_addr":"172.21.15.33:6823/29685","heartbeat_front_addr":"172.21.15.33:6821/29685","state":["exists","up"]},{"osd":1,"uuid":"7c55af6b-1aeb-49a6-94bd-3faccf2d6445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":29683},{"type":"v1","addr":"172.21.15.33:6801","nonce":29683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":29683},{"type":"v1","addr":"172.21.15.33:6803","nonce":29683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":29683},{"type":"v1","addr":"172.21.15.33:6807","nonce":29683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":29683},{"type":"v1","addr":"172.21.15.33:6805","nonce":29683}]},"public_addr":"172.21.15.33:6801/29683","cluster_addr":"172.21.15.33:6803/29683","heartbeat_back_addr":"172.21.15.33:6807/29683","heartbeat_front_addr":"172.21.15.33:6805/29683","state":["exists","up"]},{"osd":2,"uuid":"5c54b971-af63-4b5e-898a-ee8072a7524b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":29684},{"type":"v1","addr":"172.21.15.33:6809","nonce":29684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":29684},{"type":"v1","addr":"172.21.15.33:6811","nonce":29684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":29684},{"type":"v1","addr":"172.21.15.33:6815","nonce":29684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":29684},{"type":"v1","addr":"172.21.15.33:6813","nonce":29684}]},"public_addr":"172.21.15.33:6809/29684","cluster_addr":"172.21.15.33:6811/29684","heartbeat_back_addr":"172.21.15.33:6815/29684","heartbeat_front_addr":"172.21.15.33:6813/29684","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T21:55:49.936627+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T21:55:49.941640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T21:55:50.195038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T21:55:55.592 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-07T21:55:55.592 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-07T21:55:55.875 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:55.875 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":11,"fsid":"d80f57a9-566f-4802-89c6-ac2d96473541","created":"2023-12-07T21:55:44.914448+0000","modified":"2023-12-07T21:55:54.917517+0000","last_up_change":"2023-12-07T21:55:51.902623+0000","last_in_change":"2023-12-07T21:55:45.771064+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-07T21:55:51.973567+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-07T21:55:53.407039+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"dc992a6a-05d7-42e3-b943-34ea323f923c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":29685},{"type":"v1","addr":"172.21.15.33:6817","nonce":29685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":29685},{"type":"v1","addr":"172.21.15.33:6819","nonce":29685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":29685},{"type":"v1","addr":"172.21.15.33:6823","nonce":29685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":29685},{"type":"v1","addr":"172.21.15.33:6821","nonce":29685}]},"public_addr":"172.21.15.33:6817/29685","cluster_addr":"172.21.15.33:6819/29685","heartbeat_back_addr":"172.21.15.33:6823/29685","heartbeat_front_addr":"172.21.15.33:6821/29685","state":["exists","up"]},{"osd":1,"uuid":"7c55af6b-1aeb-49a6-94bd-3faccf2d6445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":29683},{"type":"v1","addr":"172.21.15.33:6801","nonce":29683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":29683},{"type":"v1","addr":"172.21.15.33:6803","nonce":29683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":29683},{"type":"v1","addr":"172.21.15.33:6807","nonce":29683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":29683},{"type":"v1","addr":"172.21.15.33:6805","nonce":29683}]},"public_addr":"172.21.15.33:6801/29683","cluster_addr":"172.21.15.33:6803/29683","heartbeat_back_addr":"172.21.15.33:6807/29683","heartbeat_front_addr":"172.21.15.33:6805/29683","state":["exists","up"]},{"osd":2,"uuid":"5c54b971-af63-4b5e-898a-ee8072a7524b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":29684},{"type":"v1","addr":"172.21.15.33:6809","nonce":29684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":29684},{"type":"v1","addr":"172.21.15.33:6811","nonce":29684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":29684},{"type":"v1","addr":"172.21.15.33:6815","nonce":29684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":29684},{"type":"v1","addr":"172.21.15.33:6813","nonce":29684}]},"public_addr":"172.21.15.33:6809/29684","cluster_addr":"172.21.15.33:6811/29684","heartbeat_back_addr":"172.21.15.33:6815/29684","heartbeat_front_addr":"172.21.15.33:6813/29684","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T21:55:49.936627+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T21:55:49.941640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T21:55:50.195038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T21:55:55.890 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-07T21:55:56.042 INFO:teuthology.orchestra.run.smithi033.stdout:34359738371 2023-12-07T21:55:56.042 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-07T21:55:56.195 INFO:teuthology.orchestra.run.smithi033.stdout:34359738370 2023-12-07T21:55:56.196 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-07T21:55:56.344 INFO:teuthology.orchestra.run.smithi033.stdout:34359738370 2023-12-07T21:55:56.345 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-07T21:55:56.628 INFO:teuthology.orchestra.run.smithi033.stdout:34359738370 2023-12-07T21:55:56.643 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738370 for osd.0 2023-12-07T21:55:57.643 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-07T21:55:57.938 INFO:teuthology.orchestra.run.smithi033.stdout:34359738371 2023-12-07T21:55:57.958 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.0 2023-12-07T21:55:57.958 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-07T21:55:58.225 INFO:teuthology.orchestra.run.smithi033.stdout:34359738370 2023-12-07T21:55:58.239 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.1 2023-12-07T21:55:58.239 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-07T21:55:58.509 INFO:teuthology.orchestra.run.smithi033.stdout:34359738370 2023-12-07T21:55:58.522 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.2 2023-12-07T21:55:58.523 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-07T21:55:58.523 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-07T21:55:58.795 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:58.798 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2023-12-07T21:55:58.810 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-12-07T21:55:56.838857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":891584,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":313681216,"statfs":{"total":322122547200,"available":321209565184,"internally_reserved":0,"allocated":303104,"data_stored":81904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-07T21:55:54.941078+0000","last_change":"2023-12-07T21:55:54.941078+0000","last_active":"2023-12-07T21:55:54.941078+0000","last_peered":"2023-12-07T21:55:54.941078+0000","last_clean":"2023-12-07T21:55:54.941078+0000","last_became_active":"2023-12-07T21:55:54.940222+0000","last_became_peered":"2023-12-07T21:55:54.940222+0000","last_unstale":"2023-12-07T21:55:54.941078+0000","last_undegraded":"2023-12-07T21:55:54.941078+0000","last_fullsized":"2023-12-07T21:55:54.941078+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_clean_scrub_stamp":"2023-12-07T21:55:53.910246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-07T21:55:54.940557+0000","last_change":"2023-12-07T21:55:54.940557+0000","last_active":"2023-12-07T21:55:54.940557+0000","last_peered":"2023-12-07T21:55:54.940557+0000","last_clean":"2023-12-07T21:55:54.940557+0000","last_became_active":"2023-12-07T21:55:54.939706+0000","last_became_peered":"2023-12-07T21:55:54.939706+0000","last_unstale":"2023-12-07T21:55:54.940557+0000","last_undegraded":"2023-12-07T21:55:54.940557+0000","last_fullsized":"2023-12-07T21:55:54.940557+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_clean_scrub_stamp":"2023-12-07T21:55:53.910246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-07T21:55:54.940659+0000","last_change":"2023-12-07T21:55:54.940659+0000","last_active":"2023-12-07T21:55:54.940659+0000","last_peered":"2023-12-07T21:55:54.940659+0000","last_clean":"2023-12-07T21:55:54.940659+0000","last_became_active":"2023-12-07T21:55:54.939711+0000","last_became_peered":"2023-12-07T21:55:54.939711+0000","last_unstale":"2023-12-07T21:55:54.940659+0000","last_undegraded":"2023-12-07T21:55:54.940659+0000","last_fullsized":"2023-12-07T21:55:54.940659+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_clean_scrub_stamp":"2023-12-07T21:55:53.910246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-07T21:55:54.940830+0000","last_change":"2023-12-07T21:55:54.940830+0000","last_active":"2023-12-07T21:55:54.940830+0000","last_peered":"2023-12-07T21:55:54.940830+0000","last_clean":"2023-12-07T21:55:54.940830+0000","last_became_active":"2023-12-07T21:55:54.939663+0000","last_became_peered":"2023-12-07T21:55:54.939663+0000","last_unstale":"2023-12-07T21:55:54.940830+0000","last_undegraded":"2023-12-07T21:55:54.940830+0000","last_fullsized":"2023-12-07T21:55:54.940830+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_clean_scrub_stamp":"2023-12-07T21:55:53.910246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-07T21:55:54.942239+0000","last_change":"2023-12-07T21:55:54.942239+0000","last_active":"2023-12-07T21:55:54.942239+0000","last_peered":"2023-12-07T21:55:54.942239+0000","last_clean":"2023-12-07T21:55:54.942239+0000","last_became_active":"2023-12-07T21:55:54.941961+0000","last_became_peered":"2023-12-07T21:55:54.941961+0000","last_unstale":"2023-12-07T21:55:54.942239+0000","last_undegraded":"2023-12-07T21:55:54.942239+0000","last_fullsized":"2023-12-07T21:55:54.942239+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_clean_scrub_stamp":"2023-12-07T21:55:53.910246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-07T21:55:54.943885+0000","last_change":"2023-12-07T21:55:54.942673+0000","last_active":"2023-12-07T21:55:54.943885+0000","last_peered":"2023-12-07T21:55:54.943885+0000","last_clean":"2023-12-07T21:55:54.943885+0000","last_became_active":"2023-12-07T21:55:54.942363+0000","last_became_peered":"2023-12-07T21:55:54.942363+0000","last_unstale":"2023-12-07T21:55:54.943885+0000","last_undegraded":"2023-12-07T21:55:54.943885+0000","last_fullsized":"2023-12-07T21:55:54.943885+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_clean_scrub_stamp":"2023-12-07T21:55:53.910246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-07T21:55:54.943519+0000","last_change":"2023-12-07T21:55:54.942404+0000","last_active":"2023-12-07T21:55:54.943519+0000","last_peered":"2023-12-07T21:55:54.943519+0000","last_clean":"2023-12-07T21:55:54.943519+0000","last_became_active":"2023-12-07T21:55:54.941636+0000","last_became_peered":"2023-12-07T21:55:54.941636+0000","last_unstale":"2023-12-07T21:55:54.943519+0000","last_undegraded":"2023-12-07T21:55:54.943519+0000","last_fullsized":"2023-12-07T21:55:54.943519+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_clean_scrub_stamp":"2023-12-07T21:55:53.910246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-07T21:55:54.941222+0000","last_change":"2023-12-07T21:55:54.941222+0000","last_active":"2023-12-07T21:55:54.941222+0000","last_peered":"2023-12-07T21:55:54.941222+0000","last_clean":"2023-12-07T21:55:54.941222+0000","last_became_active":"2023-12-07T21:55:54.940754+0000","last_became_peered":"2023-12-07T21:55:54.940754+0000","last_unstale":"2023-12-07T21:55:54.941222+0000","last_undegraded":"2023-12-07T21:55:54.941222+0000","last_fullsized":"2023-12-07T21:55:54.941222+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_clean_scrub_stamp":"2023-12-07T21:55:53.910246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-07T21:55:53.929522+0000","last_change":"2023-12-07T21:55:53.928879+0000","last_active":"2023-12-07T21:55:53.929522+0000","last_peered":"2023-12-07T21:55:53.929522+0000","last_clean":"2023-12-07T21:55:53.929522+0000","last_became_active":"2023-12-07T21:55:53.928459+0000","last_became_peered":"2023-12-07T21:55:53.928459+0000","last_unstale":"2023-12-07T21:55:53.929522+0000","last_undegraded":"2023-12-07T21:55:53.929522+0000","last_fullsized":"2023-12-07T21:55:53.929522+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:52.904771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:52.904771+0000","last_clean_scrub_stamp":"2023-12-07T21:55:52.904771+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":29654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738370,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560408,"statfs":{"total":107374182400,"available":107069857792,"internally_reserved":0,"allocated":98304,"data_stored":26125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560408,"statfs":{"total":107374182400,"available":107069857792,"internally_reserved":0,"allocated":98304,"data_stored":26125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-07T21:55:58.811 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-07T21:55:59.087 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:59.090 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2023-12-07T21:55:59.100 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2023-12-07T21:55:58.839510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":891584,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":313681216,"statfs":{"total":322122547200,"available":321209565184,"internally_reserved":0,"allocated":303104,"data_stored":81904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-07T21:55:54.941078+0000","last_change":"2023-12-07T21:55:54.941078+0000","last_active":"2023-12-07T21:55:54.941078+0000","last_peered":"2023-12-07T21:55:54.941078+0000","last_clean":"2023-12-07T21:55:54.941078+0000","last_became_active":"2023-12-07T21:55:54.940222+0000","last_became_peered":"2023-12-07T21:55:54.940222+0000","last_unstale":"2023-12-07T21:55:54.941078+0000","last_undegraded":"2023-12-07T21:55:54.941078+0000","last_fullsized":"2023-12-07T21:55:54.941078+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_clean_scrub_stamp":"2023-12-07T21:55:53.910246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-07T21:55:54.940557+0000","last_change":"2023-12-07T21:55:54.940557+0000","last_active":"2023-12-07T21:55:54.940557+0000","last_peered":"2023-12-07T21:55:54.940557+0000","last_clean":"2023-12-07T21:55:54.940557+0000","last_became_active":"2023-12-07T21:55:54.939706+0000","last_became_peered":"2023-12-07T21:55:54.939706+0000","last_unstale":"2023-12-07T21:55:54.940557+0000","last_undegraded":"2023-12-07T21:55:54.940557+0000","last_fullsized":"2023-12-07T21:55:54.940557+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_clean_scrub_stamp":"2023-12-07T21:55:53.910246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-07T21:55:54.940659+0000","last_change":"2023-12-07T21:55:54.940659+0000","last_active":"2023-12-07T21:55:54.940659+0000","last_peered":"2023-12-07T21:55:54.940659+0000","last_clean":"2023-12-07T21:55:54.940659+0000","last_became_active":"2023-12-07T21:55:54.939711+0000","last_became_peered":"2023-12-07T21:55:54.939711+0000","last_unstale":"2023-12-07T21:55:54.940659+0000","last_undegraded":"2023-12-07T21:55:54.940659+0000","last_fullsized":"2023-12-07T21:55:54.940659+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_clean_scrub_stamp":"2023-12-07T21:55:53.910246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-07T21:55:54.940830+0000","last_change":"2023-12-07T21:55:54.940830+0000","last_active":"2023-12-07T21:55:54.940830+0000","last_peered":"2023-12-07T21:55:54.940830+0000","last_clean":"2023-12-07T21:55:54.940830+0000","last_became_active":"2023-12-07T21:55:54.939663+0000","last_became_peered":"2023-12-07T21:55:54.939663+0000","last_unstale":"2023-12-07T21:55:54.940830+0000","last_undegraded":"2023-12-07T21:55:54.940830+0000","last_fullsized":"2023-12-07T21:55:54.940830+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_clean_scrub_stamp":"2023-12-07T21:55:53.910246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-07T21:55:54.942239+0000","last_change":"2023-12-07T21:55:54.942239+0000","last_active":"2023-12-07T21:55:54.942239+0000","last_peered":"2023-12-07T21:55:54.942239+0000","last_clean":"2023-12-07T21:55:54.942239+0000","last_became_active":"2023-12-07T21:55:54.941961+0000","last_became_peered":"2023-12-07T21:55:54.941961+0000","last_unstale":"2023-12-07T21:55:54.942239+0000","last_undegraded":"2023-12-07T21:55:54.942239+0000","last_fullsized":"2023-12-07T21:55:54.942239+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_clean_scrub_stamp":"2023-12-07T21:55:53.910246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-07T21:55:54.943885+0000","last_change":"2023-12-07T21:55:54.942673+0000","last_active":"2023-12-07T21:55:54.943885+0000","last_peered":"2023-12-07T21:55:54.943885+0000","last_clean":"2023-12-07T21:55:54.943885+0000","last_became_active":"2023-12-07T21:55:54.942363+0000","last_became_peered":"2023-12-07T21:55:54.942363+0000","last_unstale":"2023-12-07T21:55:54.943885+0000","last_undegraded":"2023-12-07T21:55:54.943885+0000","last_fullsized":"2023-12-07T21:55:54.943885+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_clean_scrub_stamp":"2023-12-07T21:55:53.910246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-07T21:55:54.943519+0000","last_change":"2023-12-07T21:55:54.942404+0000","last_active":"2023-12-07T21:55:54.943519+0000","last_peered":"2023-12-07T21:55:54.943519+0000","last_clean":"2023-12-07T21:55:54.943519+0000","last_became_active":"2023-12-07T21:55:54.941636+0000","last_became_peered":"2023-12-07T21:55:54.941636+0000","last_unstale":"2023-12-07T21:55:54.943519+0000","last_undegraded":"2023-12-07T21:55:54.943519+0000","last_fullsized":"2023-12-07T21:55:54.943519+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_clean_scrub_stamp":"2023-12-07T21:55:53.910246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-12-07T21:55:54.941222+0000","last_change":"2023-12-07T21:55:54.941222+0000","last_active":"2023-12-07T21:55:54.941222+0000","last_peered":"2023-12-07T21:55:54.941222+0000","last_clean":"2023-12-07T21:55:54.941222+0000","last_became_active":"2023-12-07T21:55:54.940754+0000","last_became_peered":"2023-12-07T21:55:54.940754+0000","last_unstale":"2023-12-07T21:55:54.941222+0000","last_undegraded":"2023-12-07T21:55:54.941222+0000","last_fullsized":"2023-12-07T21:55:54.941222+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:53.910246+0000","last_clean_scrub_stamp":"2023-12-07T21:55:53.910246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-12-07T21:55:53.929522+0000","last_change":"2023-12-07T21:55:53.928879+0000","last_active":"2023-12-07T21:55:53.929522+0000","last_peered":"2023-12-07T21:55:53.929522+0000","last_clean":"2023-12-07T21:55:53.929522+0000","last_became_active":"2023-12-07T21:55:53.928459+0000","last_became_peered":"2023-12-07T21:55:53.928459+0000","last_unstale":"2023-12-07T21:55:53.929522+0000","last_undegraded":"2023-12-07T21:55:53.929522+0000","last_fullsized":"2023-12-07T21:55:53.929522+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-07T21:55:52.904771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-07T21:55:52.904771+0000","last_clean_scrub_stamp":"2023-12-07T21:55:52.904771+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560400,"statfs":{"total":107374182400,"available":107069849600,"internally_reserved":0,"allocated":106496,"data_stored":29654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560408,"statfs":{"total":107374182400,"available":107069857792,"internally_reserved":0,"allocated":98304,"data_stored":26125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":104560408,"statfs":{"total":107374182400,"available":107069857792,"internally_reserved":0,"allocated":98304,"data_stored":26125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-07T21:55:59.101 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-07T21:55:59.101 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-07T21:55:59.101 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-07T21:55:59.102 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-07T21:55:59.409 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:55:59.422 INFO:teuthology.orchestra.run.smithi033.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-07T21:55:59.422 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-07T21:55:59.423 INFO:teuthology.run_tasks:Running task openssl_keys... 2023-12-07T21:55:59.435 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p /home/ubuntu/cephtest/ca 2023-12-07T21:55:59.443 DEBUG:teuthology.orchestra.run.smithi033:> openssl req -x509 -nodes -newkey rsa:4096 -keyout /home/ubuntu/cephtest/ca/kmiproot.key -days 365 -out /home/ubuntu/cephtest/ca/kmiproot.crt -subj /CN=kmiproot 2023-12-07T21:55:59.472 INFO:teuthology.orchestra.run.smithi033.stderr:Generating a RSA private key 2023-12-07T21:55:59.673 INFO:teuthology.orchestra.run.smithi033.stderr:.........................................++++ 2023-12-07T21:56:00.144 INFO:teuthology.orchestra.run.smithi033.stderr:.......................................................................................................++++ 2023-12-07T21:56:00.144 INFO:teuthology.orchestra.run.smithi033.stderr:writing new private key to '/home/ubuntu/cephtest/ca/kmiproot.key' 2023-12-07T21:56:00.144 INFO:teuthology.orchestra.run.smithi033.stderr:----- 2023-12-07T21:56:00.153 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p /home/ubuntu/cephtest/ca 2023-12-07T21:56:00.201 DEBUG:teuthology.orchestra.run.smithi033:> openssl req -nodes -newkey rsa:2048 -keyout /home/ubuntu/cephtest/ca/kmip-client.key -out /home/ubuntu/cephtest/ca/kmip-client.csr -subj /CN=rgw-client 2023-12-07T21:56:00.212 INFO:teuthology.orchestra.run.smithi033.stderr:Generating a RSA private key 2023-12-07T21:56:00.234 INFO:teuthology.orchestra.run.smithi033.stderr:...................+++++ 2023-12-07T21:56:00.257 INFO:teuthology.orchestra.run.smithi033.stderr:......................+++++ 2023-12-07T21:56:00.258 INFO:teuthology.orchestra.run.smithi033.stderr:writing new private key to '/home/ubuntu/cephtest/ca/kmip-client.key' 2023-12-07T21:56:00.258 INFO:teuthology.orchestra.run.smithi033.stderr:----- 2023-12-07T21:56:00.260 DEBUG:teuthology.orchestra.run.smithi033:> openssl x509 -req -in /home/ubuntu/cephtest/ca/kmip-client.csr -CA /home/ubuntu/cephtest/ca/kmiproot.crt -CAkey /home/ubuntu/cephtest/ca/kmiproot.key -CAcreateserial -out /home/ubuntu/cephtest/ca/kmip-client.crt -days 365 -sha256 2023-12-07T21:56:00.269 INFO:teuthology.orchestra.run.smithi033.stderr:Signature ok 2023-12-07T21:56:00.269 INFO:teuthology.orchestra.run.smithi033.stderr:subject=CN = rgw-client 2023-12-07T21:56:00.269 INFO:teuthology.orchestra.run.smithi033.stderr:Getting CA Private Key 2023-12-07T21:56:00.283 DEBUG:teuthology.orchestra.run.smithi033:> rm /home/ubuntu/cephtest/ca/kmip-client.csr /home/ubuntu/cephtest/ca/kmiproot.srl 2023-12-07T21:56:00.332 DEBUG:teuthology.orchestra.run.smithi033:> openssl verify -CAfile /home/ubuntu/cephtest/ca/kmiproot.crt /home/ubuntu/cephtest/ca/kmip-client.crt 2023-12-07T21:56:00.343 INFO:teuthology.orchestra.run.smithi033.stdout:/home/ubuntu/cephtest/ca/kmip-client.crt: OK 2023-12-07T21:56:00.344 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p /home/ubuntu/cephtest/ca 2023-12-07T21:56:00.351 DEBUG:teuthology.orchestra.run.smithi033:> openssl req -nodes -newkey rsa:2048 -keyout /home/ubuntu/cephtest/ca/kmip-server.key -out /home/ubuntu/cephtest/ca/kmip-server.csr -subj /CN=smithi033.front.sepia.ceph.com 2023-12-07T21:56:00.360 INFO:teuthology.orchestra.run.smithi033.stderr:Generating a RSA private key 2023-12-07T21:56:00.410 INFO:teuthology.orchestra.run.smithi033.stderr:................................................+++++ 2023-12-07T21:56:00.416 INFO:teuthology.orchestra.run.smithi033.stderr:....+++++ 2023-12-07T21:56:00.416 INFO:teuthology.orchestra.run.smithi033.stderr:writing new private key to '/home/ubuntu/cephtest/ca/kmip-server.key' 2023-12-07T21:56:00.416 INFO:teuthology.orchestra.run.smithi033.stderr:----- 2023-12-07T21:56:00.418 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:56:00.418 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/ca/kmip-server.ext 2023-12-07T21:56:00.465 DEBUG:teuthology.orchestra.run.smithi033:> openssl x509 -req -in /home/ubuntu/cephtest/ca/kmip-server.csr -CA /home/ubuntu/cephtest/ca/kmiproot.crt -CAkey /home/ubuntu/cephtest/ca/kmiproot.key -CAcreateserial -out /home/ubuntu/cephtest/ca/kmip-server.crt -days 365 -sha256 -extfile /home/ubuntu/cephtest/ca/kmip-server.ext 2023-12-07T21:56:00.476 INFO:teuthology.orchestra.run.smithi033.stderr:Signature ok 2023-12-07T21:56:00.477 INFO:teuthology.orchestra.run.smithi033.stderr:subject=CN = smithi033.front.sepia.ceph.com 2023-12-07T21:56:00.477 INFO:teuthology.orchestra.run.smithi033.stderr:Getting CA Private Key 2023-12-07T21:56:00.490 DEBUG:teuthology.orchestra.run.smithi033:> rm /home/ubuntu/cephtest/ca/kmip-server.csr /home/ubuntu/cephtest/ca/kmiproot.srl /home/ubuntu/cephtest/ca/kmip-server.ext 2023-12-07T21:56:00.494 DEBUG:teuthology.orchestra.run.smithi033:> openssl verify -CAfile /home/ubuntu/cephtest/ca/kmiproot.crt /home/ubuntu/cephtest/ca/kmip-server.crt 2023-12-07T21:56:00.545 INFO:teuthology.orchestra.run.smithi033.stdout:/home/ubuntu/cephtest/ca/kmip-server.crt: OK 2023-12-07T21:56:00.545 INFO:teuthology.run_tasks:Running task exec... 2023-12-07T21:56:00.555 INFO:teuthology.task.exec:Executing custom commands... 2023-12-07T21:56:00.555 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi033.front.sepia.ceph.com 2023-12-07T21:56:00.555 DEBUG:teuthology.orchestra.run.smithi033:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'chmod 644 /home/ubuntu/cephtest/ca/kmip-client.key' 2023-12-07T21:56:00.572 INFO:teuthology.run_tasks:Running task pykmip... 2023-12-07T21:56:00.584 DEBUG:tasks.pykmip:PyKMIP config is {'client.0': {'clientca': 'kmiproot', 'clientcert': 'kmip-client', 'secrets': [{'name': 'pykmip-my-key-1'}, {'name': 'pykmip-my-key-2'}], 'servercert': 'kmip-server'}} 2023-12-07T21:56:00.584 INFO:tasks.pykmip:Downloading pykmip... 2023-12-07T21:56:00.584 INFO:tasks.pykmip:Using branch 'master' for pykmip 2023-12-07T21:56:00.584 INFO:tasks.pykmip:sha1=None 2023-12-07T21:56:00.585 DEBUG:teuthology.orchestra.run.smithi033:> git clone -b master https://github.com/OpenKMIP/PyKMIP /home/ubuntu/cephtest/pykmip 2023-12-07T21:56:00.621 INFO:teuthology.orchestra.run.smithi033.stderr:Cloning into '/home/ubuntu/cephtest/pykmip'... 2023-12-07T21:56:01.252 INFO:tasks.pykmip:Setting up virtualenv for pykmip... 2023-12-07T21:56:01.252 DEBUG:teuthology.orchestra.run.smithi033:> cd /home/ubuntu/cephtest/pykmip && python3 -m venv .pykmipenv 2023-12-07T21:56:02.876 DEBUG:teuthology.orchestra.run.smithi033:> cd /home/ubuntu/cephtest/pykmip && . .pykmipenv/bin/activate && pip install --upgrade pip 2023-12-07T21:56:04.102 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting pip 2023-12-07T21:56:04.222 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading pip-23.3.1-py3-none-any.whl (2.1 MB) 2023-12-07T21:56:04.900 INFO:teuthology.orchestra.run.smithi033.stdout:Installing collected packages: pip 2023-12-07T21:56:04.900 INFO:teuthology.orchestra.run.smithi033.stdout: Attempting uninstall: pip 2023-12-07T21:56:04.900 INFO:teuthology.orchestra.run.smithi033.stdout: Found existing installation: pip 20.0.2 2023-12-07T21:56:04.962 INFO:teuthology.orchestra.run.smithi033.stdout: Uninstalling pip-20.0.2: 2023-12-07T21:56:04.964 INFO:teuthology.orchestra.run.smithi033.stdout: Successfully uninstalled pip-20.0.2 2023-12-07T21:56:05.932 INFO:teuthology.orchestra.run.smithi033.stdout:Successfully installed pip-23.3.1 2023-12-07T21:56:05.978 DEBUG:teuthology.orchestra.run.smithi033:> cd /home/ubuntu/cephtest/pykmip && . .pykmipenv/bin/activate && pip install pytz -e /home/ubuntu/cephtest/pykmip 2023-12-07T21:56:06.391 INFO:teuthology.orchestra.run.smithi033.stdout:Obtaining file:///home/ubuntu/cephtest/pykmip 2023-12-07T21:56:06.395 INFO:teuthology.orchestra.run.smithi033.stdout: Installing build dependencies: started 2023-12-07T21:56:08.819 INFO:teuthology.orchestra.run.smithi033.stdout: Installing build dependencies: finished with status 'done' 2023-12-07T21:56:08.821 INFO:teuthology.orchestra.run.smithi033.stdout: Checking if build backend supports build_editable: started 2023-12-07T21:56:08.943 INFO:teuthology.orchestra.run.smithi033.stdout: Checking if build backend supports build_editable: finished with status 'done' 2023-12-07T21:56:08.944 INFO:teuthology.orchestra.run.smithi033.stdout: Getting requirements to build editable: started 2023-12-07T21:56:09.128 INFO:teuthology.orchestra.run.smithi033.stdout: Getting requirements to build editable: finished with status 'done' 2023-12-07T21:56:09.134 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing editable metadata (pyproject.toml): started 2023-12-07T21:56:09.329 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing editable metadata (pyproject.toml): finished with status 'done' 2023-12-07T21:56:09.572 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting pytz 2023-12-07T21:56:09.653 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading pytz-2023.3.post1-py2.py3-none-any.whl.metadata (22 kB) 2023-12-07T21:56:10.037 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting cryptography (from PyKMIP==0.11.0.dev1) 2023-12-07T21:56:10.056 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading cryptography-41.0.7-cp37-abi3-manylinux_2_28_x86_64.whl.metadata (5.2 kB) 2023-12-07T21:56:10.209 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting enum-compat (from PyKMIP==0.11.0.dev1) 2023-12-07T21:56:10.228 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading enum_compat-0.0.3-py3-none-any.whl (1.3 kB) 2023-12-07T21:56:10.401 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting requests (from PyKMIP==0.11.0.dev1) 2023-12-07T21:56:10.418 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB) 2023-12-07T21:56:10.593 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting six (from PyKMIP==0.11.0.dev1) 2023-12-07T21:56:10.626 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-12-07T21:56:11.196 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting sqlalchemy (from PyKMIP==0.11.0.dev1) 2023-12-07T21:56:11.212 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading SQLAlchemy-2.0.23-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.6 kB) 2023-12-07T21:56:11.520 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting cffi>=1.12 (from cryptography->PyKMIP==0.11.0.dev1) 2023-12-07T21:56:11.539 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading cffi-1.16.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB) 2023-12-07T21:56:11.802 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting charset-normalizer<4,>=2 (from requests->PyKMIP==0.11.0.dev1) 2023-12-07T21:56:11.821 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading charset_normalizer-3.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (33 kB) 2023-12-07T21:56:11.977 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting idna<4,>=2.5 (from requests->PyKMIP==0.11.0.dev1) 2023-12-07T21:56:11.995 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading idna-3.6-py3-none-any.whl.metadata (9.9 kB) 2023-12-07T21:56:12.178 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting urllib3<3,>=1.21.1 (from requests->PyKMIP==0.11.0.dev1) 2023-12-07T21:56:12.195 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading urllib3-2.1.0-py3-none-any.whl.metadata (6.4 kB) 2023-12-07T21:56:12.353 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting certifi>=2017.4.17 (from requests->PyKMIP==0.11.0.dev1) 2023-12-07T21:56:12.371 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading certifi-2023.11.17-py3-none-any.whl.metadata (2.2 kB) 2023-12-07T21:56:12.574 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting typing-extensions>=4.2.0 (from sqlalchemy->PyKMIP==0.11.0.dev1) 2023-12-07T21:56:12.607 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading typing_extensions-4.8.0-py3-none-any.whl.metadata (3.0 kB) 2023-12-07T21:56:12.889 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting greenlet!=0.4.17 (from sqlalchemy->PyKMIP==0.11.0.dev1) 2023-12-07T21:56:12.910 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading greenlet-3.0.1-cp38-cp38-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (3.7 kB) 2023-12-07T21:56:13.080 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting pycparser (from cffi>=1.12->cryptography->PyKMIP==0.11.0.dev1) 2023-12-07T21:56:13.097 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB) 2023-12-07T21:56:13.175 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 118.7/118.7 kB 1.5 MB/s eta 0:00:00 2023-12-07T21:56:13.226 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading pytz-2023.3.post1-py2.py3-none-any.whl (502 kB) 2023-12-07T21:56:13.309 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 502.5/502.5 kB 6.3 MB/s eta 0:00:00 2023-12-07T21:56:13.340 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading cryptography-41.0.7-cp37-abi3-manylinux_2_28_x86_64.whl (4.4 MB) 2023-12-07T21:56:13.577 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 4.4/4.4 MB 18.8 MB/s eta 0:00:00 2023-12-07T21:56:13.597 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading requests-2.31.0-py3-none-any.whl (62 kB) 2023-12-07T21:56:13.652 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 62.6/62.6 kB 1.0 MB/s eta 0:00:00 2023-12-07T21:56:13.677 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading SQLAlchemy-2.0.23-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB) 2023-12-07T21:56:13.811 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 3.1/3.1 MB 23.8 MB/s eta 0:00:00 2023-12-07T21:56:13.833 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading certifi-2023.11.17-py3-none-any.whl (162 kB) 2023-12-07T21:56:13.895 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 162.5/162.5 kB 2.7 MB/s eta 0:00:00 2023-12-07T21:56:13.919 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading cffi-1.16.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (444 kB) 2023-12-07T21:56:13.987 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 444.7/444.7 kB 6.9 MB/s eta 0:00:00 2023-12-07T21:56:14.010 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading charset_normalizer-3.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (141 kB) 2023-12-07T21:56:14.071 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 141.1/141.1 kB 2.3 MB/s eta 0:00:00 2023-12-07T21:56:14.095 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading greenlet-3.0.1-cp38-cp38-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (618 kB) 2023-12-07T21:56:14.188 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 618.6/618.6 kB 6.9 MB/s eta 0:00:00 2023-12-07T21:56:14.212 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading idna-3.6-py3-none-any.whl (61 kB) 2023-12-07T21:56:14.298 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 61.6/61.6 kB 641.9 kB/s eta 0:00:00 2023-12-07T21:56:14.318 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading typing_extensions-4.8.0-py3-none-any.whl (31 kB) 2023-12-07T21:56:14.390 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading urllib3-2.1.0-py3-none-any.whl (104 kB) 2023-12-07T21:56:14.450 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 104.6/104.6 kB 1.8 MB/s eta 0:00:00 2023-12-07T21:56:14.496 INFO:teuthology.orchestra.run.smithi033.stdout:Building wheels for collected packages: PyKMIP 2023-12-07T21:56:14.497 INFO:teuthology.orchestra.run.smithi033.stdout: Building editable for PyKMIP (pyproject.toml): started 2023-12-07T21:56:14.731 INFO:teuthology.orchestra.run.smithi033.stdout: Building editable for PyKMIP (pyproject.toml): finished with status 'done' 2023-12-07T21:56:14.732 INFO:teuthology.orchestra.run.smithi033.stdout: Created wheel for PyKMIP: filename=PyKMIP-0.11.0.dev1-0.editable-py3-none-any.whl size=7833 sha256=217d7254ba66c2b70c5c3215d1783ef6a9aafbbaf53254f7bf0c0333d7c951c3 2023-12-07T21:56:14.733 INFO:teuthology.orchestra.run.smithi033.stdout: Stored in directory: /tmp/pip-ephem-wheel-cache-455xc487/wheels/ea/d8/28/3faf5b092e89e900bdd64c179a1cc2d58f85008fc572ca64af 2023-12-07T21:56:14.739 INFO:teuthology.orchestra.run.smithi033.stdout:Successfully built PyKMIP 2023-12-07T21:56:14.822 INFO:teuthology.orchestra.run.smithi033.stdout:Installing collected packages: pytz, enum-compat, urllib3, typing-extensions, six, pycparser, idna, greenlet, charset-normalizer, certifi, sqlalchemy, requests, cffi, cryptography, PyKMIP 2023-12-07T21:56:17.219 INFO:teuthology.orchestra.run.smithi033.stdout:Successfully installed PyKMIP-0.11.0.dev1 certifi-2023.11.17 cffi-1.16.0 charset-normalizer-3.3.2 cryptography-41.0.7 enum-compat-0.0.3 greenlet-3.0.1 idna-3.6 pycparser-2.21 pytz-2023.3.post1 requests-2.31.0 six-1.16.0 sqlalchemy-2.0.23 typing-extensions-4.8.0 urllib3-2.1.0 2023-12-07T21:56:17.389 INFO:tasks.pykmip:Installing system dependenies for PyKMIP... 2023-12-07T21:56:17.402 DEBUG:teuthology.orchestra.run.smithi033:> cd /home/ubuntu/cephtest/pykmip && . .pykmipenv/bin/activate && pip install bindep 2023-12-07T21:56:17.919 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting bindep 2023-12-07T21:56:17.990 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading bindep-2.11.0-py2.py3-none-any.whl (34 kB) 2023-12-07T21:56:18.168 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting Parsley (from bindep) 2023-12-07T21:56:18.188 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading Parsley-1.3-py2.py3-none-any.whl (88 kB) 2023-12-07T21:56:18.277 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 88.9/88.9 kB 978.9 kB/s eta 0:00:00 2023-12-07T21:56:18.392 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting pbr>=2.0.0 (from bindep) 2023-12-07T21:56:18.410 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading pbr-6.0.0-py2.py3-none-any.whl.metadata (1.3 kB) 2023-12-07T21:56:18.574 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting distro>=1.7.0 (from bindep) 2023-12-07T21:56:18.593 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading distro-1.8.0-py3-none-any.whl (20 kB) 2023-12-07T21:56:19.174 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting packaging (from bindep) 2023-12-07T21:56:19.191 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading packaging-23.2-py3-none-any.whl.metadata (3.2 kB) 2023-12-07T21:56:19.265 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading pbr-6.0.0-py2.py3-none-any.whl (107 kB) 2023-12-07T21:56:19.369 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 107.5/107.5 kB 1.0 MB/s eta 0:00:00 2023-12-07T21:56:19.384 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading packaging-23.2-py3-none-any.whl (53 kB) 2023-12-07T21:56:19.448 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 53.0/53.0 kB 719.1 kB/s eta 0:00:00 2023-12-07T21:56:19.541 INFO:teuthology.orchestra.run.smithi033.stdout:Installing collected packages: Parsley, pbr, packaging, distro, bindep 2023-12-07T21:56:20.018 INFO:teuthology.orchestra.run.smithi033.stdout:Successfully installed Parsley-1.3 bindep-2.11.0 distro-1.8.0 packaging-23.2 pbr-6.0.0 2023-12-07T21:56:20.098 DEBUG:teuthology.orchestra.run.smithi033:> cd /home/ubuntu/cephtest/pykmip && . .pykmipenv/bin/activate && bindep --brief --file /tmp/bindep-d26zbshv.txt 2023-12-07T21:56:20.740 INFO:teuthology.orchestra.run.smithi033.stdout:libffi-dev 2023-12-07T21:56:20.740 INFO:teuthology.orchestra.run.smithi033.stdout:libsqlite3-dev 2023-12-07T21:56:20.740 INFO:teuthology.orchestra.run.smithi033.stdout:libssl-dev 2023-12-07T21:56:20.741 INFO:teuthology.orchestra.run.smithi033.stdout:python-dev 2023-12-07T21:56:20.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T21:56:20.908 INFO:teuthology.packaging:Installing package libffi-dev on ubuntu@smithi033.front.sepia.ceph.com 2023-12-07T21:56:20.909 DEBUG:teuthology.orchestra.run.smithi033:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install libffi-dev 2023-12-07T21:56:20.979 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-07T21:56:21.190 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-12-07T21:56:21.190 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-12-07T21:56:21.380 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-12-07T21:56:21.381 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-07T21:56:21.381 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-07T21:56:21.414 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2023-12-07T21:56:21.415 INFO:teuthology.orchestra.run.smithi033.stdout: libffi-dev 2023-12-07T21:56:21.617 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 1 newly installed, 0 to remove and 259 not upgraded. 2023-12-07T21:56:21.617 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 57.0 kB of archives. 2023-12-07T21:56:21.617 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 312 kB of additional disk space will be used. 2023-12-07T21:56:21.617 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libffi-dev amd64 3.3-4 [57.0 kB] 2023-12-07T21:56:22.189 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 57.0 kB in 0s (149 kB/s) 2023-12-07T21:56:22.303 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libffi-dev:amd64. 2023-12-07T21:56:22.362 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 149177 files and directories currently installed.) 2023-12-07T21:56:22.366 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../libffi-dev_3.3-4_amd64.deb ... 2023-12-07T21:56:22.409 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libffi-dev:amd64 (3.3-4) ... 2023-12-07T21:56:22.862 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libffi-dev:amd64 (3.3-4) ... 2023-12-07T21:56:22.991 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2023-12-07T21:56:23.560 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-12-07T21:56:25.183 INFO:teuthology.orchestra.run.smithi033.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-07T21:56:25.186 INFO:teuthology.packaging:Installing package libsqlite3-dev on ubuntu@smithi033.front.sepia.ceph.com 2023-12-07T21:56:25.186 DEBUG:teuthology.orchestra.run.smithi033:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install libsqlite3-dev 2023-12-07T21:56:25.257 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-07T21:56:25.467 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-12-07T21:56:25.467 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-12-07T21:56:25.670 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-12-07T21:56:25.671 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-07T21:56:25.671 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-07T21:56:25.671 INFO:teuthology.orchestra.run.smithi033.stdout:The following additional packages will be installed: 2023-12-07T21:56:25.672 INFO:teuthology.orchestra.run.smithi033.stdout: libsqlite3-0 2023-12-07T21:56:25.673 INFO:teuthology.orchestra.run.smithi033.stdout:Suggested packages: 2023-12-07T21:56:25.673 INFO:teuthology.orchestra.run.smithi033.stdout: sqlite3-doc 2023-12-07T21:56:25.708 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2023-12-07T21:56:25.708 INFO:teuthology.orchestra.run.smithi033.stdout: libsqlite3-dev 2023-12-07T21:56:25.709 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages will be upgraded: 2023-12-07T21:56:25.710 INFO:teuthology.orchestra.run.smithi033.stdout: libsqlite3-0 2023-12-07T21:56:25.763 INFO:teuthology.orchestra.run.smithi033.stdout:1 upgraded, 1 newly installed, 0 to remove and 258 not upgraded. 2023-12-07T21:56:25.763 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 1,245 kB of archives. 2023-12-07T21:56:25.763 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 2,372 kB of additional disk space will be used. 2023-12-07T21:56:25.763 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libsqlite3-0 amd64 3.31.1-4ubuntu0.5 [549 kB] 2023-12-07T21:56:25.875 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libsqlite3-dev amd64 3.31.1-4ubuntu0.5 [697 kB] 2023-12-07T21:56:26.272 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 1,245 kB in 0s (7,050 kB/s) 2023-12-07T21:56:26.520 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 149211 files and directories currently installed.) 2023-12-07T21:56:26.524 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../libsqlite3-0_3.31.1-4ubuntu0.5_amd64.deb ... 2023-12-07T21:56:26.689 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libsqlite3-0:amd64 (3.31.1-4ubuntu0.5) over (3.31.1-4ubuntu0.3) ... 2023-12-07T21:56:27.077 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libsqlite3-dev:amd64. 2023-12-07T21:56:27.094 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../libsqlite3-dev_3.31.1-4ubuntu0.5_amd64.deb ... 2023-12-07T21:56:27.135 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libsqlite3-dev:amd64 (3.31.1-4ubuntu0.5) ... 2023-12-07T21:56:27.510 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libsqlite3-0:amd64 (3.31.1-4ubuntu0.5) ... 2023-12-07T21:56:27.606 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libsqlite3-dev:amd64 (3.31.1-4ubuntu0.5) ... 2023-12-07T21:56:27.766 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-12-07T21:56:28.201 INFO:teuthology.orchestra.run.smithi033.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-07T21:56:28.204 INFO:teuthology.packaging:Installing package libssl-dev on ubuntu@smithi033.front.sepia.ceph.com 2023-12-07T21:56:28.204 DEBUG:teuthology.orchestra.run.smithi033:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install libssl-dev 2023-12-07T21:56:28.276 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-07T21:56:28.487 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-12-07T21:56:28.488 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-12-07T21:56:28.687 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-12-07T21:56:28.687 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-07T21:56:28.687 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-07T21:56:28.688 INFO:teuthology.orchestra.run.smithi033.stdout:The following additional packages will be installed: 2023-12-07T21:56:28.689 INFO:teuthology.orchestra.run.smithi033.stdout: libssl1.1 2023-12-07T21:56:28.690 INFO:teuthology.orchestra.run.smithi033.stdout:Suggested packages: 2023-12-07T21:56:28.690 INFO:teuthology.orchestra.run.smithi033.stdout: libssl-doc 2023-12-07T21:56:28.724 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2023-12-07T21:56:28.725 INFO:teuthology.orchestra.run.smithi033.stdout: libssl-dev 2023-12-07T21:56:28.728 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages will be upgraded: 2023-12-07T21:56:28.728 INFO:teuthology.orchestra.run.smithi033.stdout: libssl1.1 2023-12-07T21:56:28.785 INFO:teuthology.orchestra.run.smithi033.stdout:1 upgraded, 1 newly installed, 0 to remove and 257 not upgraded. 2023-12-07T21:56:28.786 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 2,906 kB of archives. 2023-12-07T21:56:28.786 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 8,011 kB of additional disk space will be used. 2023-12-07T21:56:28.786 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.20 [1,321 kB] 2023-12-07T21:56:28.941 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.20 [1,584 kB] 2023-12-07T21:56:29.282 INFO:teuthology.orchestra.run.smithi033.stdout:Preconfiguring packages ... 2023-12-07T21:56:29.440 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 2,906 kB in 0s (12.7 MB/s) 2023-12-07T21:56:29.748 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 149220 files and directories currently installed.) 2023-12-07T21:56:29.753 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../libssl1.1_1.1.1f-1ubuntu2.20_amd64.deb ... 2023-12-07T21:56:29.930 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.20) over (1.1.1f-1ubuntu2.16) ... 2023-12-07T21:56:30.359 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-12-07T21:56:30.378 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../libssl-dev_1.1.1f-1ubuntu2.20_amd64.deb ... 2023-12-07T21:56:30.417 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.20) ... 2023-12-07T21:56:30.895 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.20) ... 2023-12-07T21:56:31.223 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.20) ... 2023-12-07T21:56:31.350 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-12-07T21:56:31.835 INFO:teuthology.orchestra.run.smithi033.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-07T21:56:31.837 INFO:teuthology.packaging:Installing package python-dev on ubuntu@smithi033.front.sepia.ceph.com 2023-12-07T21:56:31.838 DEBUG:teuthology.orchestra.run.smithi033:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev 2023-12-07T21:56:31.913 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-07T21:56:32.125 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-12-07T21:56:32.126 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-12-07T21:56:32.316 INFO:teuthology.orchestra.run.smithi033.stdout:python-dev-is-python2 is already the newest version (2.7.17-4). 2023-12-07T21:56:32.316 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-12-07T21:56:32.317 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-07T21:56:32.317 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-07T21:56:32.360 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 0 newly installed, 0 to remove and 257 not upgraded. 2023-12-07T21:56:32.360 INFO:teuthology.orchestra.run.smithi033.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-07T21:56:32.362 DEBUG:teuthology.orchestra.run.smithi033:> cd /home/ubuntu/cephtest/pykmip && cp /home/ubuntu/cephtest/pykmip/examples/policy.json /home/ubuntu/cephtest/pykmip 2023-12-07T21:56:32.366 INFO:tasks.pykmip:#0 cclient='client.0' cconfig={'clientca': 'kmiproot', 'clientcert': 'kmip-client', 'secrets': [{'name': 'pykmip-my-key-1'}, {'name': 'pykmip-my-key-2'}], 'servercert': 'kmip-server'} 2023-12-07T21:56:32.367 INFO:tasks.pykmip:#1 ip,p,h 172.21.15.33 5696 smithi033.front.sepia.ceph.com 2023-12-07T21:56:32.367 INFO:tasks.pykmip:#2 clientca kmiproot 2023-12-07T21:56:32.367 INFO:tasks.pykmip:#3 servercert kmip-server 2023-12-07T21:56:32.367 INFO:tasks.pykmip:#4 servercert Namespace(certificate='/home/ubuntu/cephtest/ca/kmip-server.crt', client='client.0', key='/home/ubuntu/cephtest/ca/kmip-server.key', key_type='rsa:2048', name='kmip-server', remote=Remote(name='ubuntu@smithi033.front.sepia.ceph.com')) 2023-12-07T21:56:32.367 INFO:tasks.pykmip:#3 clientcert kmip-client 2023-12-07T21:56:32.367 INFO:tasks.pykmip:#4 clientcert Namespace(certificate='/home/ubuntu/cephtest/ca/kmip-client.crt', client='client.0', key='/home/ubuntu/cephtest/ca/kmip-client.key', key_type='rsa:2048', name='kmip-client', remote=Remote(name='ubuntu@smithi033.front.sepia.ceph.com')) 2023-12-07T21:56:32.367 INFO:tasks.pykmip:#5 clientca Namespace(certificate='/home/ubuntu/cephtest/ca/kmiproot.crt', client='client.0', key='/home/ubuntu/cephtest/ca/kmiproot.key', key_type='rsa:4096', name='kmiproot', remote=Remote(name='ubuntu@smithi033.front.sepia.ceph.com')) 2023-12-07T21:56:32.367 INFO:tasks.pykmip:#6 serverkey /home/ubuntu/cephtest/ca/kmip-server.key servercert /home/ubuntu/cephtest/ca/kmip-server.crt 2023-12-07T21:56:32.367 INFO:tasks.pykmip:#6 clientkey /home/ubuntu/cephtest/ca/kmip-client.key clientcert /home/ubuntu/cephtest/ca/kmip-client.crt 2023-12-07T21:56:32.367 INFO:tasks.pykmip:#7 clientca /home/ubuntu/cephtest/ca/kmiproot.crt 2023-12-07T21:56:32.376 INFO:tasks.pykmip:Running pykmip... 2023-12-07T21:56:32.376 INFO:tasks.pykmip.client.0:Restarting daemon 2023-12-07T21:56:32.376 DEBUG:teuthology.orchestra.run.smithi033:> (cd /home/ubuntu/cephtest/pykmip && exec bash -c 'cd /home/ubuntu/cephtest/pykmip && . .pykmipenv/bin/activate && HOME=/home/ubuntu/cephtest/pykmip && exec pykmip-server -f pykmip.conf -l /home/ubuntu/cephtest/pykmip/pykmip.log & { read; kill %1; }') 2023-12-07T21:56:32.382 INFO:tasks.pykmip.client.0:Started 2023-12-07T21:56:42.384 DEBUG:teuthology.orchestra.run.smithi033:> . cephtest/pykmip/.pykmipenv/bin/activate;python 2023-12-07T21:56:43.106 INFO:teuthology.orchestra.run.smithi033.stdout:[{"Unique Identifier": "1", "Name": "pykmip-my-key-1", "Object Type": "ObjectType.SYMMETRIC_KEY", "Cryptographic Algorithm": "CryptographicAlgorithm.AES", "Cryptographic Length": "256", "Operation Policy Name": "default", "Cryptographic Usage Mask": "12", "State": "State.ACTIVE", "Initial Date": "Thu Dec 7 21:56:42 2023"}, {"Unique Identifier": "2", "Name": "pykmip-my-key-2", "Object Type": "ObjectType.SYMMETRIC_KEY", "Cryptographic Algorithm": "CryptographicAlgorithm.AES", "Cryptographic Length": "256", "Operation Policy Name": "default", "Cryptographic Usage Mask": "12", "State": "State.ACTIVE", "Initial Date": "Thu Dec 7 21:56:42 2023"}] 2023-12-07T21:56:43.182 INFO:teuthology.run_tasks:Running task rgw... 2023-12-07T21:56:43.207 DEBUG:tasks.rgw:config is {'client.0': {'use-pykmip-role': 'client.0'}} 2023-12-07T21:56:43.207 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2023-12-07T21:56:43.207 INFO:tasks.rgw:Creating data pools 2023-12-07T21:56:43.208 DEBUG:tasks.rgw:Obtaining remote for client client.0 2023-12-07T21:56:43.208 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2023-12-07T21:56:43.936 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'default.rgw.buckets.data' created 2023-12-07T21:56:43.951 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2023-12-07T21:56:44.971 INFO:teuthology.orchestra.run.smithi033.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-12-07T21:56:44.981 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2023-12-07T21:56:45.958 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'default.rgw.buckets.index' created 2023-12-07T21:56:45.972 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2023-12-07T21:56:46.983 INFO:teuthology.orchestra.run.smithi033.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-12-07T21:56:47.001 DEBUG:tasks.rgw:Pools created 2023-12-07T21:56:47.001 INFO:tasks.rgw:Starting rgw... 2023-12-07T21:56:47.002 INFO:tasks.rgw:rgw client.0 config is {'use-pykmip-role': 'client.0'} 2023-12-07T21:56:47.002 INFO:tasks.rgw:Using civetweb as radosgw frontend 2023-12-07T21:56:47.002 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /home/ubuntu/cephtest/ca/kmip-client.crt /etc/ceph/ 2023-12-07T21:56:47.010 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /home/ubuntu/cephtest/ca/kmip-client.key /etc/ceph/ 2023-12-07T21:56:47.018 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /home/ubuntu/cephtest/ca/kmip-server.crt /etc/ceph/ 2023-12-07T21:56:47.027 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /home/ubuntu/cephtest/ca/kmip-server.key /etc/ceph/ 2023-12-07T21:56:47.036 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /home/ubuntu/cephtest/ca/kmiproot.key /etc/ceph/ 2023-12-07T21:56:47.095 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /home/ubuntu/cephtest/ca/kmiproot.crt /etc/ceph/ 2023-12-07T21:56:47.105 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 600 /etc/ceph/kmip-client.crt /etc/ceph/kmip-client.key /etc/ceph/kmip-server.crt /etc/ceph/kmip-server.key /etc/ceph/kmiproot.key /etc/ceph/kmiproot.crt 2023-12-07T21:56:47.116 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown ceph /etc/ceph/kmip-client.crt /etc/ceph/kmip-client.key /etc/ceph/kmip-server.crt /etc/ceph/kmip-server.key /etc/ceph/kmiproot.key /etc/ceph/kmiproot.crt 2023-12-07T21:56:47.131 INFO:tasks.rgw.client.0:Restarting daemon 2023-12-07T21:56:47.131 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_kmip_addr 172.21.15.33:5696 --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2023-12-07T21:56:47.177 INFO:tasks.rgw.client.0:Started 2023-12-07T21:56:47.177 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi033.front.sepia.ceph.com:80/ 2023-12-07T21:56:47.177 DEBUG:teuthology.orchestra.run.smithi033:> curl http://smithi033.front.sepia.ceph.com:80/ 2023-12-07T21:56:47.224 INFO:teuthology.orchestra.run.smithi033.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-12-07T21:56:47.224 INFO:teuthology.orchestra.run.smithi033.stderr: Dload Upload Total Spent Left Speed 2023-12-07T21:56:47.225 INFO:teuthology.orchestra.run.smithi033.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-12-07T21:56:47.225 INFO:teuthology.orchestra.run.smithi033.stderr:curl: (7) Failed to connect to smithi033.front.sepia.ceph.com port 80: Connection refused 2023-12-07T21:56:47.225 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-12-07T21:56:48.226 DEBUG:teuthology.orchestra.run.smithi033:> curl http://smithi033.front.sepia.ceph.com:80/ 2023-12-07T21:56:48.243 INFO:teuthology.orchestra.run.smithi033.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-12-07T21:56:48.243 INFO:teuthology.orchestra.run.smithi033.stderr: Dload Upload Total Spent Left Speed 2023-12-07T21:56:48.244 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-12-07T21:56:48.245 INFO:teuthology.orchestra.run.smithi033.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-12-07T21:56:48.245 INFO:teuthology.orchestra.run.smithi033.stderr:curl: (7) Failed to connect to smithi033.front.sepia.ceph.com port 80: Connection refused 2023-12-07T21:56:50.245 DEBUG:teuthology.orchestra.run.smithi033:> curl http://smithi033.front.sepia.ceph.com:80/ 2023-12-07T21:56:50.262 INFO:teuthology.orchestra.run.smithi033.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-12-07T21:56:50.262 INFO:teuthology.orchestra.run.smithi033.stderr: Dload Upload Total Spent Left Speed 2023-12-07T21:56:50.263 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-12-07T21:56:50.263 INFO:teuthology.orchestra.run.smithi033.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-12-07T21:56:50.263 INFO:teuthology.orchestra.run.smithi033.stderr:curl: (7) Failed to connect to smithi033.front.sepia.ceph.com port 80: Connection refused 2023-12-07T21:56:54.265 DEBUG:teuthology.orchestra.run.smithi033:> curl http://smithi033.front.sepia.ceph.com:80/ 2023-12-07T21:56:54.282 INFO:teuthology.orchestra.run.smithi033.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-12-07T21:56:54.282 INFO:teuthology.orchestra.run.smithi033.stderr: Dload Upload Total Spent Left Speed 2023-12-07T21:56:54.283 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-12-07T21:56:54.283 INFO:teuthology.orchestra.run.smithi033.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-12-07T21:56:54.283 INFO:teuthology.orchestra.run.smithi033.stderr:curl: (7) Failed to connect to smithi033.front.sepia.ceph.com port 80: Connection refused 2023-12-07T21:56:56.145 INFO:tasks.rgw.client.0.smithi033.stdout:2023-12-07T21:56:56.140+0000 7f1f6f6e0cc0 -1 IMPORTANT: the civetweb frontend is now deprecated and will be removed in a future release 2023-12-07T21:57:02.285 DEBUG:teuthology.orchestra.run.smithi033:> curl http://smithi033.front.sepia.ceph.com:80/ 2023-12-07T21:57:02.302 INFO:teuthology.orchestra.run.smithi033.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-12-07T21:57:02.302 INFO:teuthology.orchestra.run.smithi033.stderr: Dload Upload Total Spent Left Speed 2023-12-07T21:57:02.305 INFO:teuthology.orchestra.run.smithi033.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-12-07T21:57:02.306 INFO:teuthology.orchestra.run.smithi033.stdout:anonymous 2023-12-07T21:57:02.306 INFO:teuthology.run_tasks:Running task s3tests... 2023-12-07T21:57:02.321 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-pacific', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}} 2023-12-07T21:57:02.321 INFO:tasks.s3tests:Downloading s3-tests... 2023-12-07T21:57:02.321 INFO:tasks.s3tests:Using branch 'ceph-pacific' for s3tests 2023-12-07T21:57:02.321 DEBUG:teuthology.orchestra.run.smithi033:> git clone -b ceph-pacific https://git.ceph.com/s3-tests.git /home/ubuntu/cephtest/s3-tests 2023-12-07T21:57:02.352 INFO:teuthology.orchestra.run.smithi033.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests'... 2023-12-07T21:57:03.161 INFO:tasks.s3tests:Creating rgw users... 2023-12-07T21:57:03.162 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2023-12-07T21:57:03.162 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key JZGLQKOZONUJRVXRXMQY --secret j5VyIYCDRIfWnVwfDXtD39//q8Dun2qubN2Hh/qfsqpNASQsycnOYg== --email foo.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-12-07T21:57:03.247 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setuser ceph since I am not root 2023-12-07T21:57:03.247 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setgroup ceph since I am not root 2023-12-07T21:57:03.258 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.252+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.260 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.256+0000 7feb0b8b5c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.260 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.256+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.261 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.256+0000 7feb0b8b5c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.261 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.256+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.261 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.256+0000 7feb0b8b5c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.261 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.256+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.261 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.256+0000 7feb0b8b5c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:03.280 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.276+0000 7feb0b8b5c80 20 RGWRados::pool_iterate: got zonegroup_info.8b14f025-a9cb-4535-8f2a-c4795b716a58 2023-12-07T21:57:03.281 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.276+0000 7feb0b8b5c80 20 RGWRados::pool_iterate: got zone_info.5de1c49b-6b66-4f92-9f67-9eb1162319b3 2023-12-07T21:57:03.281 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.276+0000 7feb0b8b5c80 20 RGWRados::pool_iterate: got zone_names.default 2023-12-07T21:57:03.281 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.276+0000 7feb0b8b5c80 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-07T21:57:03.281 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.276+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.281 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.276+0000 7feb0b8b5c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:03.281 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.276+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.282 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.276+0000 7feb0b8b5c80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T21:57:03.282 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.276+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.282 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.276+0000 7feb0b8b5c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:03.282 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.276+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.282 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.276+0000 7feb0b8b5c80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T21:57:03.282 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.276+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.283 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.276+0000 7feb0b8b5c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.284 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.276+0000 7feb0b8b5c80 10 cannot find current period zonegroup using local zonegroup 2023-12-07T21:57:03.284 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.276+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.284 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.284 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.284 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:03.284 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.285 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T21:57:03.285 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 20 zonegroup default 2023-12-07T21:57:03.285 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.285 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.285 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 10 Cannot find current period zone using local zone 2023-12-07T21:57:03.285 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.286 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.286 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.286 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:03.286 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.287 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T21:57:03.287 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 20 zone default found 2023-12-07T21:57:03.287 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.280+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.288 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.284+0000 7feb0b8b5c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.288 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.284+0000 7feb0b8b5c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.288 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.284+0000 7feb0b8b5c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.288 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.284+0000 7feb0b8b5c80 20 started sync module instance, tier type = 2023-12-07T21:57:03.288 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.284+0000 7feb0b8b5c80 20 started zone id=5de1c49b-6b66-4f92-9f67-9eb1162319b3 (name=default) with tier type = 2023-12-07T21:57:03.310 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.304+0000 7feb0b8b5c80 20 add_watcher() i=0 2023-12-07T21:57:03.310 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.304+0000 7feb0b8b5c80 20 add_watcher() i=1 2023-12-07T21:57:03.310 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.304+0000 7feb0b8b5c80 20 add_watcher() i=2 2023-12-07T21:57:03.310 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.304+0000 7feb0b8b5c80 20 add_watcher() i=3 2023-12-07T21:57:03.310 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.304+0000 7feb0b8b5c80 20 add_watcher() i=4 2023-12-07T21:57:03.310 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.304+0000 7feb0b8b5c80 20 add_watcher() i=5 2023-12-07T21:57:03.310 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.304+0000 7feb0b8b5c80 20 add_watcher() i=6 2023-12-07T21:57:03.311 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.304+0000 7feb0b8b5c80 20 add_watcher() i=7 2023-12-07T21:57:03.311 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.308+0000 7feb0b8b5c80 2 all 8 watchers are set, enabling cache 2023-12-07T21:57:03.314 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.308+0000 7fead9feb700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T21:57:03.314 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.308+0000 7feb0b8b5c80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-12-07T21:57:03.314 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.308+0000 7feb0b8b5c80 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-07T21:57:03.321 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.316+0000 7feb0b8b5c80 5 note: GC not initialized 2023-12-07T21:57:03.324 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7fea50ff9700 20 reqs_thread_entry: start 2023-12-07T21:57:03.325 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7fea467fc700 20 reqs_thread_entry: start 2023-12-07T21:57:03.325 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 20 init_complete bucket index max shards: 11 2023-12-07T21:57:03.325 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 10 rgw notify: Started notification manager with: 1 workers 2023-12-07T21:57:03.325 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7fea44ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Dec 7 21:57:33 2023 2023-12-07T21:57:03.325 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T21:57:03.325 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 20 get_system_obj_state: rctx=0x7ffca0a96428 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55bd0e7fe020 s->prefetch_data=0 2023-12-07T21:57:03.325 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-12-07T21:57:03.325 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7fea2bfff700 20 reqs_thread_entry: start 2023-12-07T21:57:03.326 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2023-12-07T21:57:03.326 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-12-07T21:57:03.326 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 20 get_system_obj_state: rctx=0x7ffca0a96438 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x55bd0e7fe020 s->prefetch_data=0 2023-12-07T21:57:03.326 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2023-12-07T21:57:03.326 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2023-12-07T21:57:03.326 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-12-07T21:57:03.326 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 20 get_system_obj_state: rctx=0x7ffca0a96438 obj=default.rgw.meta:users.keys:JZGLQKOZONUJRVXRXMQY state=0x55bd0e7fe020 s->prefetch_data=0 2023-12-07T21:57:03.326 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 10 cache get: name=default.rgw.meta+users.keys+JZGLQKOZONUJRVXRXMQY : miss 2023-12-07T21:57:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 10 cache put: name=default.rgw.meta+users.keys+JZGLQKOZONUJRVXRXMQY info.flags=0x0 2023-12-07T21:57:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 10 adding default.rgw.meta+users.keys+JZGLQKOZONUJRVXRXMQY to cache LRU end 2023-12-07T21:57:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 20 get_system_obj_state: rctx=0x7ffca0a95ce8 obj=default.rgw.meta:users.keys:JZGLQKOZONUJRVXRXMQY state=0x55bd0e7fe020 s->prefetch_data=0 2023-12-07T21:57:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 10 cache get: name=default.rgw.meta+users.keys+JZGLQKOZONUJRVXRXMQY : hit (negative entry) 2023-12-07T21:57:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 20 get_system_obj_state: rctx=0x7ffca0a96068 obj=default.rgw.meta:users.keys:JZGLQKOZONUJRVXRXMQY state=0x55bd0e7fe020 s->prefetch_data=0 2023-12-07T21:57:03.327 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.320+0000 7feb0b8b5c80 10 cache get: name=default.rgw.meta+users.keys+JZGLQKOZONUJRVXRXMQY : hit (negative entry) 2023-12-07T21:57:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.324+0000 7feb0b8b5c80 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-12-07T21:57:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.324+0000 7feb0b8b5c80 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-12-07T21:57:03.329 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.324+0000 7feb0b8b5c80 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-12-07T21:57:03.330 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.324+0000 7feb02ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182400 cookie 94270481765040 notifier 4197 bl.length()=640 2023-12-07T21:57:03.330 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.324+0000 7feb02ffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-12-07T21:57:03.330 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.324+0000 7feb02ffd700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-12-07T21:57:03.333 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.328+0000 7feb0b8b5c80 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-12-07T21:57:03.334 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.328+0000 7feb0b8b5c80 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-12-07T21:57:03.334 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.328+0000 7feb0b8b5c80 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-12-07T21:57:03.334 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.328+0000 7feb02ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182400 cookie 94270481786768 notifier 4197 bl.length()=194 2023-12-07T21:57:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.328+0000 7feb02ffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-12-07T21:57:03.335 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.328+0000 7feb02ffd700 10 rgw watcher librados: moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-12-07T21:57:03.338 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.332+0000 7feb0b8b5c80 10 cache put: name=default.rgw.meta+users.keys+JZGLQKOZONUJRVXRXMQY info.flags=0x7 2023-12-07T21:57:03.338 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.332+0000 7feb0b8b5c80 10 moving default.rgw.meta+users.keys+JZGLQKOZONUJRVXRXMQY to cache LRU end 2023-12-07T21:57:03.338 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.332+0000 7feb0b8b5c80 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:JZGLQKOZONUJRVXRXMQY, ofs0, ns] 2023-12-07T21:57:03.338 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.332+0000 7feb02ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 107374182400 cookie 94270481772416 notifier 4197 bl.length()=186 2023-12-07T21:57:03.338 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.332+0000 7feb02ffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+JZGLQKOZONUJRVXRXMQY info.flags=0x7 2023-12-07T21:57:03.338 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.332+0000 7feb02ffd700 10 rgw watcher librados: moving default.rgw.meta+users.keys+JZGLQKOZONUJRVXRXMQY to cache LRU end 2023-12-07T21:57:03.339 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-12-07T21:57:03.339 INFO:teuthology.orchestra.run.smithi033.stdout: "user_id": "foo.client.0", 2023-12-07T21:57:03.339 INFO:teuthology.orchestra.run.smithi033.stdout: "display_name": "Mr. foo.client.0", 2023-12-07T21:57:03.339 INFO:teuthology.orchestra.run.smithi033.stdout: "email": "foo.client.0+test@test.test", 2023-12-07T21:57:03.340 INFO:teuthology.orchestra.run.smithi033.stdout: "suspended": 0, 2023-12-07T21:57:03.340 INFO:teuthology.orchestra.run.smithi033.stdout: "max_buckets": 1000, 2023-12-07T21:57:03.340 INFO:teuthology.orchestra.run.smithi033.stdout: "subusers": [], 2023-12-07T21:57:03.340 INFO:teuthology.orchestra.run.smithi033.stdout: "keys": [ 2023-12-07T21:57:03.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-12-07T21:57:03.340 INFO:teuthology.orchestra.run.smithi033.stdout: "user": "foo.client.0", 2023-12-07T21:57:03.340 INFO:teuthology.orchestra.run.smithi033.stdout: "access_key": "JZGLQKOZONUJRVXRXMQY", 2023-12-07T21:57:03.340 INFO:teuthology.orchestra.run.smithi033.stdout: "secret_key": "j5VyIYCDRIfWnVwfDXtD39//q8Dun2qubN2Hh/qfsqpNASQsycnOYg==" 2023-12-07T21:57:03.340 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-12-07T21:57:03.340 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-12-07T21:57:03.340 INFO:teuthology.orchestra.run.smithi033.stdout: "swift_keys": [], 2023-12-07T21:57:03.340 INFO:teuthology.orchestra.run.smithi033.stdout: "caps": [ 2023-12-07T21:57:03.340 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-12-07T21:57:03.340 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "user-policy", 2023-12-07T21:57:03.340 INFO:teuthology.orchestra.run.smithi033.stdout: "perm": "*" 2023-12-07T21:57:03.340 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-12-07T21:57:03.341 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-12-07T21:57:03.341 INFO:teuthology.orchestra.run.smithi033.stdout: "op_mask": "read, write, delete", 2023-12-07T21:57:03.341 INFO:teuthology.orchestra.run.smithi033.stdout: "default_placement": "", 2023-12-07T21:57:03.341 INFO:teuthology.orchestra.run.smithi033.stdout: "default_storage_class": "", 2023-12-07T21:57:03.341 INFO:teuthology.orchestra.run.smithi033.stdout: "placement_tags": [], 2023-12-07T21:57:03.341 INFO:teuthology.orchestra.run.smithi033.stdout: "bucket_quota": { 2023-12-07T21:57:03.341 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": false, 2023-12-07T21:57:03.341 INFO:teuthology.orchestra.run.smithi033.stdout: "check_on_raw": false, 2023-12-07T21:57:03.341 INFO:teuthology.orchestra.run.smithi033.stdout: "max_size": -1, 2023-12-07T21:57:03.341 INFO:teuthology.orchestra.run.smithi033.stdout: "max_size_kb": 0, 2023-12-07T21:57:03.341 INFO:teuthology.orchestra.run.smithi033.stdout: "max_objects": -1 2023-12-07T21:57:03.341 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-12-07T21:57:03.341 INFO:teuthology.orchestra.run.smithi033.stdout: "user_quota": { 2023-12-07T21:57:03.341 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": false, 2023-12-07T21:57:03.341 INFO:teuthology.orchestra.run.smithi033.stdout: "check_on_raw": false, 2023-12-07T21:57:03.341 INFO:teuthology.orchestra.run.smithi033.stdout: "max_size": -1, 2023-12-07T21:57:03.342 INFO:teuthology.orchestra.run.smithi033.stdout: "max_size_kb": 0, 2023-12-07T21:57:03.342 INFO:teuthology.orchestra.run.smithi033.stdout: "max_objects": -1 2023-12-07T21:57:03.342 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-12-07T21:57:03.342 INFO:teuthology.orchestra.run.smithi033.stdout: "temp_url_keys": [], 2023-12-07T21:57:03.342 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "rgw", 2023-12-07T21:57:03.342 INFO:teuthology.orchestra.run.smithi033.stdout: "mfa_ids": [] 2023-12-07T21:57:03.342 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-12-07T21:57:03.342 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:57:03.343 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.340+0000 7feb0b8b5c80 20 remove_watcher() i=0 2023-12-07T21:57:03.343 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.340+0000 7feb0b8b5c80 2 removed watcher, disabling cache 2023-12-07T21:57:03.346 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.340+0000 7feb0b8b5c80 20 remove_watcher() i=1 2023-12-07T21:57:03.349 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.344+0000 7feb0b8b5c80 20 remove_watcher() i=2 2023-12-07T21:57:03.351 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.348+0000 7feb0b8b5c80 20 remove_watcher() i=3 2023-12-07T21:57:03.354 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.348+0000 7feb0b8b5c80 20 remove_watcher() i=4 2023-12-07T21:57:03.356 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.352+0000 7feb0b8b5c80 20 remove_watcher() i=5 2023-12-07T21:57:03.359 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.352+0000 7feb0b8b5c80 20 remove_watcher() i=6 2023-12-07T21:57:03.361 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.356+0000 7feb0b8b5c80 20 remove_watcher() i=7 2023-12-07T21:57:03.362 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.356+0000 7fead9feb700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T21:57:03.371 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 3850995993 --totp-seed FJOWNOEVIYJPZLPOEATK5DJISQ44FM2ZUCH2DR7BONZ5MURTA2UMBEGUZ5CBGPJ7 --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-12-07T21:57:03.417 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setuser ceph since I am not root 2023-12-07T21:57:03.417 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setgroup ceph since I am not root 2023-12-07T21:57:03.432 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.428+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.433 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.428+0000 7f1135fc7c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.433 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.428+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.434 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.428+0000 7f1135fc7c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.434 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.428+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.434 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.428+0000 7f1135fc7c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.434 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.428+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.435 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.428+0000 7f1135fc7c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:03.454 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.448+0000 7f1135fc7c80 20 RGWRados::pool_iterate: got zonegroup_info.8b14f025-a9cb-4535-8f2a-c4795b716a58 2023-12-07T21:57:03.455 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.448+0000 7f1135fc7c80 20 RGWRados::pool_iterate: got zone_info.5de1c49b-6b66-4f92-9f67-9eb1162319b3 2023-12-07T21:57:03.455 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.448+0000 7f1135fc7c80 20 RGWRados::pool_iterate: got zone_names.default 2023-12-07T21:57:03.455 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.448+0000 7f1135fc7c80 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-07T21:57:03.455 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.448+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.455 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.448+0000 7f1135fc7c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:03.456 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.448+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.456 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.452+0000 7f1135fc7c80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T21:57:03.456 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.452+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.456 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.452+0000 7f1135fc7c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:03.456 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.452+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.452+0000 7f1135fc7c80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T21:57:03.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.452+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.452+0000 7f1135fc7c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.452+0000 7f1135fc7c80 10 cannot find current period zonegroup using local zonegroup 2023-12-07T21:57:03.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.452+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.459 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.452+0000 7f1135fc7c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.459 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.452+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.459 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.452+0000 7f1135fc7c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:03.459 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.452+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.459 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.456+0000 7f1135fc7c80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T21:57:03.460 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.456+0000 7f1135fc7c80 20 zonegroup default 2023-12-07T21:57:03.460 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.456+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.460 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.456+0000 7f1135fc7c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.460 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.456+0000 7f1135fc7c80 10 Cannot find current period zone using local zone 2023-12-07T21:57:03.460 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.456+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.461 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.456+0000 7f1135fc7c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.461 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.456+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.461 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.456+0000 7f1135fc7c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:03.461 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.456+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.462 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.456+0000 7f1135fc7c80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T21:57:03.463 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.456+0000 7f1135fc7c80 20 zone default found 2023-12-07T21:57:03.463 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.456+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.463 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.456+0000 7f1135fc7c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.463 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.456+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.463 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.460+0000 7f1135fc7c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:03.463 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.460+0000 7f1135fc7c80 20 started sync module instance, tier type = 2023-12-07T21:57:03.463 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.460+0000 7f1135fc7c80 20 started zone id=5de1c49b-6b66-4f92-9f67-9eb1162319b3 (name=default) with tier type = 2023-12-07T21:57:03.492 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.488+0000 7f1135fc7c80 20 add_watcher() i=0 2023-12-07T21:57:03.492 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.488+0000 7f1135fc7c80 20 add_watcher() i=1 2023-12-07T21:57:03.492 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.488+0000 7f1135fc7c80 20 add_watcher() i=2 2023-12-07T21:57:03.492 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.488+0000 7f1135fc7c80 20 add_watcher() i=3 2023-12-07T21:57:03.492 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.488+0000 7f1135fc7c80 20 add_watcher() i=4 2023-12-07T21:57:03.492 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.488+0000 7f1135fc7c80 20 add_watcher() i=5 2023-12-07T21:57:03.492 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.488+0000 7f1135fc7c80 20 add_watcher() i=6 2023-12-07T21:57:03.493 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.488+0000 7f1135fc7c80 20 add_watcher() i=7 2023-12-07T21:57:03.493 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.488+0000 7f1135fc7c80 2 all 8 watchers are set, enabling cache 2023-12-07T21:57:03.497 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.492+0000 7f11057e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T21:57:03.497 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.492+0000 7f1135fc7c80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-12-07T21:57:03.497 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.492+0000 7f1135fc7c80 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-07T21:57:03.506 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.500+0000 7f1135fc7c80 5 note: GC not initialized 2023-12-07T21:57:03.508 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.504+0000 7f1073fff700 20 reqs_thread_entry: start 2023-12-07T21:57:03.509 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.504+0000 7f1071ffb700 20 reqs_thread_entry: start 2023-12-07T21:57:03.509 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.504+0000 7f1135fc7c80 20 init_complete bucket index max shards: 11 2023-12-07T21:57:03.509 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.504+0000 7f1135fc7c80 10 rgw notify: Started notification manager with: 1 workers 2023-12-07T21:57:03.509 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.504+0000 7f105ffff700 20 rgw notify: INFO: next queues processing will happen at: Thu Dec 7 21:57:33 2023 2023-12-07T21:57:03.509 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T21:57:03.509 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.504+0000 7f105f7fe700 20 reqs_thread_entry: start 2023-12-07T21:57:03.510 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.504+0000 7f1135fc7c80 20 get_system_obj_state: rctx=0x7ffdb24e59b8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x5634101a4020 s->prefetch_data=0 2023-12-07T21:57:03.510 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.504+0000 7f1135fc7c80 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-12-07T21:57:03.510 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.504+0000 7f1135fc7c80 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-12-07T21:57:03.510 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.504+0000 7f1135fc7c80 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-12-07T21:57:03.510 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.504+0000 7f1135fc7c80 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-07T21:57:03.510 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.504+0000 7f1135fc7c80 20 get_system_obj_state: s->obj_tag was set empty 2023-12-07T21:57:03.510 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.504+0000 7f1135fc7c80 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x11, cached=0x16) 2023-12-07T21:57:03.511 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.504+0000 7f1135fc7c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:03.511 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.508+0000 7f1135fc7c80 20 rados_obj.operate() r=0 bl.length=455 2023-12-07T21:57:03.511 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.508+0000 7f1135fc7c80 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x11 2023-12-07T21:57:03.511 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:03.508+0000 7f1135fc7c80 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-12-07T21:57:04.973 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:04.968+0000 7f1135fc7c80 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-12-07T21:57:04.973 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:04.968+0000 7f1135fc7c80 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-12-07T21:57:04.973 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:04.968+0000 7f1135fc7c80 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-12-07T21:57:04.974 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:04.968+0000 7f11313f0700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116993 cookie 94781609767840 notifier 4203 bl.length()=654 2023-12-07T21:57:04.974 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:04.968+0000 7f11313f0700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-12-07T21:57:04.974 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:04.968+0000 7f11313f0700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-12-07T21:57:04.979 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:04.972+0000 7f1135fc7c80 20 remove_watcher() i=0 2023-12-07T21:57:04.979 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:04.972+0000 7f1135fc7c80 2 removed watcher, disabling cache 2023-12-07T21:57:04.979 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:04.972+0000 7f1135fc7c80 20 remove_watcher() i=1 2023-12-07T21:57:04.983 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:04.972+0000 7f1135fc7c80 20 remove_watcher() i=2 2023-12-07T21:57:04.983 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:04.976+0000 7f1135fc7c80 20 remove_watcher() i=3 2023-12-07T21:57:04.983 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:04.976+0000 7f1135fc7c80 20 remove_watcher() i=4 2023-12-07T21:57:04.983 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:04.976+0000 7f1135fc7c80 20 remove_watcher() i=5 2023-12-07T21:57:04.985 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:04.980+0000 7f1135fc7c80 20 remove_watcher() i=6 2023-12-07T21:57:04.986 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:04.980+0000 7f1135fc7c80 20 remove_watcher() i=7 2023-12-07T21:57:04.986 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:04.980+0000 7f11057e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T21:57:04.993 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2023-12-07T21:57:04.993 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --access-key WDEERDXDFEFVEGNXAPHG --secret poNnYe6vTz6+a96oXqgH7+7La7t/OnRqw3v3S2RkaZI1X0i/xId05Q== --email bar.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-12-07T21:57:05.032 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setuser ceph since I am not root 2023-12-07T21:57:05.032 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setgroup ceph since I am not root 2023-12-07T21:57:05.046 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.040+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.047 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.044+0000 7f9af894bc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.047 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.044+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.048 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.044+0000 7f9af894bc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.048 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.044+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.048 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.044+0000 7f9af894bc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.048 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.044+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.048 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.044+0000 7f9af894bc80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.064+0000 7f9af894bc80 20 RGWRados::pool_iterate: got zonegroup_info.8b14f025-a9cb-4535-8f2a-c4795b716a58 2023-12-07T21:57:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.064+0000 7f9af894bc80 20 RGWRados::pool_iterate: got zone_info.5de1c49b-6b66-4f92-9f67-9eb1162319b3 2023-12-07T21:57:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.064+0000 7f9af894bc80 20 RGWRados::pool_iterate: got zone_names.default 2023-12-07T21:57:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.064+0000 7f9af894bc80 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-07T21:57:05.069 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.064+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.064+0000 7f9af894bc80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.064+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.064+0000 7f9af894bc80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T21:57:05.070 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.064+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.064+0000 7f9af894bc80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.064+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.071 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.068+0000 7f9af894bc80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T21:57:05.072 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.068+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.072 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.068+0000 7f9af894bc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.072 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.068+0000 7f9af894bc80 10 cannot find current period zonegroup using local zonegroup 2023-12-07T21:57:05.072 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.068+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.072 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.068+0000 7f9af894bc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.073 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.068+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.073 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.068+0000 7f9af894bc80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.073 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.068+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.074 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.068+0000 7f9af894bc80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T21:57:05.074 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.068+0000 7f9af894bc80 20 zonegroup default 2023-12-07T21:57:05.074 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.068+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.074 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.068+0000 7f9af894bc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.068+0000 7f9af894bc80 10 Cannot find current period zone using local zone 2023-12-07T21:57:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.068+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.068+0000 7f9af894bc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.075 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.072+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.076 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.072+0000 7f9af894bc80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.076 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.072+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.076 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.072+0000 7f9af894bc80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T21:57:05.077 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.072+0000 7f9af894bc80 20 zone default found 2023-12-07T21:57:05.077 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.072+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.077 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.072+0000 7f9af894bc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.077 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.072+0000 7f9af894bc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.078 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.072+0000 7f9af894bc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.078 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.072+0000 7f9af894bc80 20 started sync module instance, tier type = 2023-12-07T21:57:05.078 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.072+0000 7f9af894bc80 20 started zone id=5de1c49b-6b66-4f92-9f67-9eb1162319b3 (name=default) with tier type = 2023-12-07T21:57:05.098 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.092+0000 7f9af894bc80 20 add_watcher() i=0 2023-12-07T21:57:05.098 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.092+0000 7f9af894bc80 20 add_watcher() i=1 2023-12-07T21:57:05.098 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.092+0000 7f9af894bc80 20 add_watcher() i=2 2023-12-07T21:57:05.098 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.092+0000 7f9af894bc80 20 add_watcher() i=3 2023-12-07T21:57:05.100 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.092+0000 7f9af894bc80 20 add_watcher() i=4 2023-12-07T21:57:05.100 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.092+0000 7f9af894bc80 20 add_watcher() i=5 2023-12-07T21:57:05.100 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.092+0000 7f9af894bc80 20 add_watcher() i=6 2023-12-07T21:57:05.100 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.096+0000 7f9af894bc80 20 add_watcher() i=7 2023-12-07T21:57:05.101 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.096+0000 7f9af894bc80 2 all 8 watchers are set, enabling cache 2023-12-07T21:57:05.102 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.096+0000 7f9ac7fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T21:57:05.103 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.096+0000 7f9af894bc80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-12-07T21:57:05.103 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.096+0000 7f9af894bc80 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-07T21:57:05.110 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.104+0000 7f9af894bc80 5 note: GC not initialized 2023-12-07T21:57:05.112 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9a36ffd700 20 reqs_thread_entry: start 2023-12-07T21:57:05.113 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9a34ff9700 20 reqs_thread_entry: start 2023-12-07T21:57:05.113 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 20 init_complete bucket index max shards: 11 2023-12-07T21:57:05.113 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 10 rgw notify: Started notification manager with: 1 workers 2023-12-07T21:57:05.113 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9a2affd700 20 rgw notify: INFO: next queues processing will happen at: Thu Dec 7 21:57:35 2023 2023-12-07T21:57:05.113 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T21:57:05.113 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9a2a7fc700 20 reqs_thread_entry: start 2023-12-07T21:57:05.113 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 20 get_system_obj_state: rctx=0x7ffe5113dae8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x561079ed4020 s->prefetch_data=0 2023-12-07T21:57:05.113 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-12-07T21:57:05.114 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2023-12-07T21:57:05.114 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-12-07T21:57:05.114 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 20 get_system_obj_state: rctx=0x7ffe5113daf8 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x561079ed4020 s->prefetch_data=0 2023-12-07T21:57:05.114 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2023-12-07T21:57:05.115 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2023-12-07T21:57:05.115 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-12-07T21:57:05.115 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 20 get_system_obj_state: rctx=0x7ffe5113daf8 obj=default.rgw.meta:users.keys:WDEERDXDFEFVEGNXAPHG state=0x561079ed4020 s->prefetch_data=0 2023-12-07T21:57:05.115 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 10 cache get: name=default.rgw.meta+users.keys+WDEERDXDFEFVEGNXAPHG : miss 2023-12-07T21:57:05.115 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 10 cache put: name=default.rgw.meta+users.keys+WDEERDXDFEFVEGNXAPHG info.flags=0x0 2023-12-07T21:57:05.115 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 10 adding default.rgw.meta+users.keys+WDEERDXDFEFVEGNXAPHG to cache LRU end 2023-12-07T21:57:05.115 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 20 get_system_obj_state: rctx=0x7ffe5113d3a8 obj=default.rgw.meta:users.keys:WDEERDXDFEFVEGNXAPHG state=0x561079ed4020 s->prefetch_data=0 2023-12-07T21:57:05.115 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 10 cache get: name=default.rgw.meta+users.keys+WDEERDXDFEFVEGNXAPHG : hit (negative entry) 2023-12-07T21:57:05.115 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 20 get_system_obj_state: rctx=0x7ffe5113d728 obj=default.rgw.meta:users.keys:WDEERDXDFEFVEGNXAPHG state=0x561079ed4020 s->prefetch_data=0 2023-12-07T21:57:05.115 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.108+0000 7f9af894bc80 10 cache get: name=default.rgw.meta+users.keys+WDEERDXDFEFVEGNXAPHG : hit (negative entry) 2023-12-07T21:57:05.117 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.112+0000 7f9af894bc80 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-12-07T21:57:05.117 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.112+0000 7f9af894bc80 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-12-07T21:57:05.117 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.112+0000 7f9af894bc80 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-12-07T21:57:05.118 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.112+0000 7f9aeffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116993 cookie 94628766393568 notifier 4209 bl.length()=640 2023-12-07T21:57:05.118 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.112+0000 7f9aeffff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-12-07T21:57:05.118 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.112+0000 7f9aeffff700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-12-07T21:57:05.122 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.116+0000 7f9af894bc80 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-12-07T21:57:05.122 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.116+0000 7f9af894bc80 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-12-07T21:57:05.122 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.116+0000 7f9af894bc80 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-12-07T21:57:05.122 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.116+0000 7f9aeffff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116994 cookie 94628766393568 notifier 4209 bl.length()=194 2023-12-07T21:57:05.122 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.116+0000 7f9aeffff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-12-07T21:57:05.122 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.116+0000 7f9aeffff700 10 rgw watcher librados: moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-12-07T21:57:05.126 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.120+0000 7f9af894bc80 10 cache put: name=default.rgw.meta+users.keys+WDEERDXDFEFVEGNXAPHG info.flags=0x7 2023-12-07T21:57:05.126 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.120+0000 7f9af894bc80 10 moving default.rgw.meta+users.keys+WDEERDXDFEFVEGNXAPHG to cache LRU end 2023-12-07T21:57:05.126 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.120+0000 7f9af894bc80 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.keys:WDEERDXDFEFVEGNXAPHG, ofs0, ns] 2023-12-07T21:57:05.126 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.120+0000 7f9aef7fe700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116995 cookie 94628766393568 notifier 4209 bl.length()=186 2023-12-07T21:57:05.127 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.120+0000 7f9aef7fe700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+WDEERDXDFEFVEGNXAPHG info.flags=0x7 2023-12-07T21:57:05.127 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.120+0000 7f9aef7fe700 10 rgw watcher librados: moving default.rgw.meta+users.keys+WDEERDXDFEFVEGNXAPHG to cache LRU end 2023-12-07T21:57:05.127 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-12-07T21:57:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "user_id": "bar.client.0", 2023-12-07T21:57:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "display_name": "Mr. bar.client.0", 2023-12-07T21:57:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "email": "bar.client.0+test@test.test", 2023-12-07T21:57:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "suspended": 0, 2023-12-07T21:57:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "max_buckets": 1000, 2023-12-07T21:57:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "subusers": [], 2023-12-07T21:57:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "keys": [ 2023-12-07T21:57:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-12-07T21:57:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "user": "bar.client.0", 2023-12-07T21:57:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "access_key": "WDEERDXDFEFVEGNXAPHG", 2023-12-07T21:57:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "secret_key": "poNnYe6vTz6+a96oXqgH7+7La7t/OnRqw3v3S2RkaZI1X0i/xId05Q==" 2023-12-07T21:57:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-12-07T21:57:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-12-07T21:57:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "swift_keys": [], 2023-12-07T21:57:05.128 INFO:teuthology.orchestra.run.smithi033.stdout: "caps": [ 2023-12-07T21:57:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-12-07T21:57:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "user-policy", 2023-12-07T21:57:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "perm": "*" 2023-12-07T21:57:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-12-07T21:57:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-12-07T21:57:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "op_mask": "read, write, delete", 2023-12-07T21:57:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "default_placement": "", 2023-12-07T21:57:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "default_storage_class": "", 2023-12-07T21:57:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "placement_tags": [], 2023-12-07T21:57:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "bucket_quota": { 2023-12-07T21:57:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": false, 2023-12-07T21:57:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "check_on_raw": false, 2023-12-07T21:57:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "max_size": -1, 2023-12-07T21:57:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "max_size_kb": 0, 2023-12-07T21:57:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: "max_objects": -1 2023-12-07T21:57:05.129 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-12-07T21:57:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "user_quota": { 2023-12-07T21:57:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": false, 2023-12-07T21:57:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "check_on_raw": false, 2023-12-07T21:57:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "max_size": -1, 2023-12-07T21:57:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "max_size_kb": 0, 2023-12-07T21:57:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "max_objects": -1 2023-12-07T21:57:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-12-07T21:57:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "temp_url_keys": [], 2023-12-07T21:57:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "rgw", 2023-12-07T21:57:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: "mfa_ids": [] 2023-12-07T21:57:05.130 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-12-07T21:57:05.130 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:57:05.131 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.128+0000 7f9af894bc80 20 remove_watcher() i=0 2023-12-07T21:57:05.131 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.128+0000 7f9af894bc80 2 removed watcher, disabling cache 2023-12-07T21:57:05.134 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.128+0000 7f9af894bc80 20 remove_watcher() i=1 2023-12-07T21:57:05.136 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.132+0000 7f9af894bc80 20 remove_watcher() i=2 2023-12-07T21:57:05.138 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.132+0000 7f9af894bc80 20 remove_watcher() i=3 2023-12-07T21:57:05.141 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.136+0000 7f9af894bc80 20 remove_watcher() i=4 2023-12-07T21:57:05.143 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.140+0000 7f9af894bc80 20 remove_watcher() i=5 2023-12-07T21:57:05.146 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.140+0000 7f9af894bc80 20 remove_watcher() i=6 2023-12-07T21:57:05.149 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.144+0000 7f9af894bc80 20 remove_watcher() i=7 2023-12-07T21:57:05.149 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.144+0000 7f9ac7fdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T21:57:05.159 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 9233168855 --totp-seed TSW7VUH6DADXIMQT74SMBXKH7SH2BGBDA5AWWIS7HIYHK3T3ZNQ4EW5S3ETICCJC --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-12-07T21:57:05.206 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setuser ceph since I am not root 2023-12-07T21:57:05.206 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setgroup ceph since I am not root 2023-12-07T21:57:05.218 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.212+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.220 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.216+0000 7fd81d5cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.220 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.216+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.221 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.216+0000 7fd81d5cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.221 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.216+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.221 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.216+0000 7fd81d5cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.221 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.216+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.222 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.216+0000 7fd81d5cdc80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.241 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.236+0000 7fd81d5cdc80 20 RGWRados::pool_iterate: got zonegroup_info.8b14f025-a9cb-4535-8f2a-c4795b716a58 2023-12-07T21:57:05.241 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.236+0000 7fd81d5cdc80 20 RGWRados::pool_iterate: got zone_info.5de1c49b-6b66-4f92-9f67-9eb1162319b3 2023-12-07T21:57:05.241 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.236+0000 7fd81d5cdc80 20 RGWRados::pool_iterate: got zone_names.default 2023-12-07T21:57:05.241 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.236+0000 7fd81d5cdc80 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-07T21:57:05.241 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.236+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.242 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.236+0000 7fd81d5cdc80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.242 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.236+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.242 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.236+0000 7fd81d5cdc80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T21:57:05.242 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.236+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.242 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.236+0000 7fd81d5cdc80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.242 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.236+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.243 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T21:57:05.243 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.244 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.244 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 10 cannot find current period zonegroup using local zonegroup 2023-12-07T21:57:05.244 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.244 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.245 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.245 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.245 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.246 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T21:57:05.246 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 20 zonegroup default 2023-12-07T21:57:05.246 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.246 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.246 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 10 Cannot find current period zone using local zone 2023-12-07T21:57:05.246 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.247 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.247 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.240+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.247 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.244+0000 7fd81d5cdc80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.247 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.244+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.248 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.244+0000 7fd81d5cdc80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T21:57:05.248 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.244+0000 7fd81d5cdc80 20 zone default found 2023-12-07T21:57:05.248 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.244+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.249 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.244+0000 7fd81d5cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.249 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.244+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.249 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.244+0000 7fd81d5cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.249 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.244+0000 7fd81d5cdc80 20 started sync module instance, tier type = 2023-12-07T21:57:05.250 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.244+0000 7fd81d5cdc80 20 started zone id=5de1c49b-6b66-4f92-9f67-9eb1162319b3 (name=default) with tier type = 2023-12-07T21:57:05.283 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.276+0000 7fd81d5cdc80 20 add_watcher() i=0 2023-12-07T21:57:05.283 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.276+0000 7fd81d5cdc80 20 add_watcher() i=1 2023-12-07T21:57:05.283 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.276+0000 7fd81d5cdc80 20 add_watcher() i=2 2023-12-07T21:57:05.283 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.276+0000 7fd81d5cdc80 20 add_watcher() i=3 2023-12-07T21:57:05.283 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.276+0000 7fd81d5cdc80 20 add_watcher() i=4 2023-12-07T21:57:05.283 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.276+0000 7fd81d5cdc80 20 add_watcher() i=5 2023-12-07T21:57:05.283 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.276+0000 7fd81d5cdc80 20 add_watcher() i=6 2023-12-07T21:57:05.285 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.280+0000 7fd81d5cdc80 20 add_watcher() i=7 2023-12-07T21:57:05.285 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.280+0000 7fd81d5cdc80 2 all 8 watchers are set, enabling cache 2023-12-07T21:57:05.288 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.284+0000 7fd7ecfe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T21:57:05.288 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.284+0000 7fd81d5cdc80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-12-07T21:57:05.288 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.284+0000 7fd81d5cdc80 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-07T21:57:05.297 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.292+0000 7fd81d5cdc80 5 note: GC not initialized 2023-12-07T21:57:05.302 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd75b7fe700 20 reqs_thread_entry: start 2023-12-07T21:57:05.302 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd7597fa700 20 reqs_thread_entry: start 2023-12-07T21:57:05.302 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd81d5cdc80 20 init_complete bucket index max shards: 11 2023-12-07T21:57:05.302 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd81d5cdc80 10 rgw notify: Started notification manager with: 1 workers 2023-12-07T21:57:05.302 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd7477fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Dec 7 21:57:35 2023 2023-12-07T21:57:05.302 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T21:57:05.303 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd746ffd700 20 reqs_thread_entry: start 2023-12-07T21:57:05.303 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd81d5cdc80 20 get_system_obj_state: rctx=0x7ffe8593b318 obj=default.rgw.meta:users.uid:bar.client.0 state=0x56243b92f020 s->prefetch_data=0 2023-12-07T21:57:05.303 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd81d5cdc80 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-12-07T21:57:05.303 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd81d5cdc80 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-12-07T21:57:05.303 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd81d5cdc80 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-12-07T21:57:05.303 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd81d5cdc80 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-07T21:57:05.303 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd81d5cdc80 20 get_system_obj_state: s->obj_tag was set empty 2023-12-07T21:57:05.303 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd81d5cdc80 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x11, cached=0x16) 2023-12-07T21:57:05.303 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd81d5cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.303 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd81d5cdc80 20 rados_obj.operate() r=0 bl.length=455 2023-12-07T21:57:05.303 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd81d5cdc80 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x11 2023-12-07T21:57:05.304 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.296+0000 7fd81d5cdc80 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-12-07T21:57:05.310 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.304+0000 7fd81d5cdc80 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-12-07T21:57:05.310 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.304+0000 7fd81d5cdc80 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-12-07T21:57:05.310 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.304+0000 7fd81d5cdc80 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-12-07T21:57:05.311 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.304+0000 7fd8191f7700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116996 cookie 94713619624704 notifier 4215 bl.length()=654 2023-12-07T21:57:05.311 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.308+0000 7fd8191f7700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-12-07T21:57:05.311 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.308+0000 7fd8191f7700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-12-07T21:57:05.316 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.312+0000 7fd81d5cdc80 20 remove_watcher() i=0 2023-12-07T21:57:05.316 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.312+0000 7fd81d5cdc80 2 removed watcher, disabling cache 2023-12-07T21:57:05.318 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.312+0000 7fd81d5cdc80 20 remove_watcher() i=1 2023-12-07T21:57:05.320 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.316+0000 7fd81d5cdc80 20 remove_watcher() i=2 2023-12-07T21:57:05.323 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.316+0000 7fd81d5cdc80 20 remove_watcher() i=3 2023-12-07T21:57:05.325 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.320+0000 7fd81d5cdc80 20 remove_watcher() i=4 2023-12-07T21:57:05.328 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.324+0000 7fd81d5cdc80 20 remove_watcher() i=5 2023-12-07T21:57:05.330 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.324+0000 7fd81d5cdc80 20 remove_watcher() i=6 2023-12-07T21:57:05.332 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.328+0000 7fd81d5cdc80 20 remove_watcher() i=7 2023-12-07T21:57:05.333 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.328+0000 7fd7ecfe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T21:57:05.342 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2023-12-07T21:57:05.342 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --access-key JYGIAADVMWOHZERHZHDD --secret jjrEygxpYD95OtwOh+ljxP2xvhewdwNwcMProRlJPwT1Za+DuAr4Jw== --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --cluster ceph 2023-12-07T21:57:05.389 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setuser ceph since I am not root 2023-12-07T21:57:05.389 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setgroup ceph since I am not root 2023-12-07T21:57:05.403 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.400+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.405 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.400+0000 7fcafaca1c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.405 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.400+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.405 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.400+0000 7fcafaca1c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.405 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.400+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.405 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.400+0000 7fcafaca1c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.405 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.400+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.406 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.400+0000 7fcafaca1c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.424 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.420+0000 7fcafaca1c80 20 RGWRados::pool_iterate: got zonegroup_info.8b14f025-a9cb-4535-8f2a-c4795b716a58 2023-12-07T21:57:05.425 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.420+0000 7fcafaca1c80 20 RGWRados::pool_iterate: got zone_info.5de1c49b-6b66-4f92-9f67-9eb1162319b3 2023-12-07T21:57:05.425 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.420+0000 7fcafaca1c80 20 RGWRados::pool_iterate: got zone_names.default 2023-12-07T21:57:05.425 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.420+0000 7fcafaca1c80 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-07T21:57:05.425 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.420+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.426 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.420+0000 7fcafaca1c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.426 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.420+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.426 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.420+0000 7fcafaca1c80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T21:57:05.426 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.420+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.426 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.420+0000 7fcafaca1c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.426 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.420+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.427 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.420+0000 7fcafaca1c80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T21:57:05.427 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.420+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.427 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.428 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 10 cannot find current period zonegroup using local zonegroup 2023-12-07T21:57:05.428 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.428 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.428 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.428 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.428 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.429 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T21:57:05.429 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 20 zonegroup default 2023-12-07T21:57:05.429 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.430 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.430 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 10 Cannot find current period zone using local zone 2023-12-07T21:57:05.430 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.430 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.430 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.431 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.431 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.424+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.431 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.428+0000 7fcafaca1c80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T21:57:05.432 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.428+0000 7fcafaca1c80 20 zone default found 2023-12-07T21:57:05.432 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.428+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.432 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.428+0000 7fcafaca1c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.432 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.428+0000 7fcafaca1c80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.433 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.428+0000 7fcafaca1c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.433 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.428+0000 7fcafaca1c80 20 started sync module instance, tier type = 2023-12-07T21:57:05.433 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.428+0000 7fcafaca1c80 20 started zone id=5de1c49b-6b66-4f92-9f67-9eb1162319b3 (name=default) with tier type = 2023-12-07T21:57:05.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.452+0000 7fcafaca1c80 20 add_watcher() i=0 2023-12-07T21:57:05.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.452+0000 7fcafaca1c80 20 add_watcher() i=1 2023-12-07T21:57:05.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.452+0000 7fcafaca1c80 20 add_watcher() i=2 2023-12-07T21:57:05.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.452+0000 7fcafaca1c80 20 add_watcher() i=3 2023-12-07T21:57:05.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.452+0000 7fcafaca1c80 20 add_watcher() i=4 2023-12-07T21:57:05.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.452+0000 7fcafaca1c80 20 add_watcher() i=5 2023-12-07T21:57:05.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.452+0000 7fcafaca1c80 20 add_watcher() i=6 2023-12-07T21:57:05.459 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.456+0000 7fcafaca1c80 20 add_watcher() i=7 2023-12-07T21:57:05.459 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.456+0000 7fcafaca1c80 2 all 8 watchers are set, enabling cache 2023-12-07T21:57:05.462 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.456+0000 7fcaca7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T21:57:05.462 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.456+0000 7fcafaca1c80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-12-07T21:57:05.462 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.456+0000 7fcafaca1c80 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-07T21:57:05.470 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.464+0000 7fcafaca1c80 5 note: GC not initialized 2023-12-07T21:57:05.473 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.468+0000 7fca32ffd700 20 reqs_thread_entry: start 2023-12-07T21:57:05.473 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.468+0000 7fcafaca1c80 20 init_complete bucket index max shards: 11 2023-12-07T21:57:05.473 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.468+0000 7fca397fa700 20 reqs_thread_entry: start 2023-12-07T21:57:05.474 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.468+0000 7fcafaca1c80 10 rgw notify: Started notification manager with: 1 workers 2023-12-07T21:57:05.474 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.468+0000 7fca30ff9700 20 reqs_thread_entry: start 2023-12-07T21:57:05.474 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.468+0000 7fcafaca1c80 20 get_system_obj_state: rctx=0x7ffc4dab0828 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55ac4ac5d020 s->prefetch_data=0 2023-12-07T21:57:05.474 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.468+0000 7fcafaca1c80 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-12-07T21:57:05.474 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.468+0000 7fca317fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Dec 7 21:57:35 2023 2023-12-07T21:57:05.474 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T21:57:05.475 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.468+0000 7fcafaca1c80 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2023-12-07T21:57:05.475 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.468+0000 7fcafaca1c80 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-12-07T21:57:05.475 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.468+0000 7fcafaca1c80 20 get_system_obj_state: rctx=0x7ffc4dab0838 obj=default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test state=0x55ac4ac5d020 s->prefetch_data=0 2023-12-07T21:57:05.475 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.468+0000 7fcafaca1c80 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2023-12-07T21:57:05.475 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.468+0000 7fcafaca1c80 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2023-12-07T21:57:05.475 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.468+0000 7fcafaca1c80 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-12-07T21:57:05.475 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.468+0000 7fcafaca1c80 20 get_system_obj_state: rctx=0x7ffc4dab0838 obj=default.rgw.meta:users.keys:JYGIAADVMWOHZERHZHDD state=0x55ac4ac5d020 s->prefetch_data=0 2023-12-07T21:57:05.475 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.468+0000 7fcafaca1c80 10 cache get: name=default.rgw.meta+users.keys+JYGIAADVMWOHZERHZHDD : miss 2023-12-07T21:57:05.476 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.472+0000 7fcafaca1c80 10 cache put: name=default.rgw.meta+users.keys+JYGIAADVMWOHZERHZHDD info.flags=0x0 2023-12-07T21:57:05.476 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.472+0000 7fcafaca1c80 10 adding default.rgw.meta+users.keys+JYGIAADVMWOHZERHZHDD to cache LRU end 2023-12-07T21:57:05.476 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.472+0000 7fcafaca1c80 20 get_system_obj_state: rctx=0x7ffc4dab00e8 obj=default.rgw.meta:users.keys:JYGIAADVMWOHZERHZHDD state=0x55ac4ac5d020 s->prefetch_data=0 2023-12-07T21:57:05.476 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.472+0000 7fcafaca1c80 10 cache get: name=default.rgw.meta+users.keys+JYGIAADVMWOHZERHZHDD : hit (negative entry) 2023-12-07T21:57:05.476 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.472+0000 7fcafaca1c80 20 get_system_obj_state: rctx=0x7ffc4dab0468 obj=default.rgw.meta:users.keys:JYGIAADVMWOHZERHZHDD state=0x55ac4ac5d020 s->prefetch_data=0 2023-12-07T21:57:05.476 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.472+0000 7fcafaca1c80 10 cache get: name=default.rgw.meta+users.keys+JYGIAADVMWOHZERHZHDD : hit (negative entry) 2023-12-07T21:57:05.478 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.472+0000 7fcafaca1c80 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-12-07T21:57:05.478 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.472+0000 7fcafaca1c80 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-12-07T21:57:05.478 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.472+0000 7fcafaca1c80 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-12-07T21:57:05.479 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.472+0000 7fcaf68cb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116994 cookie 94198478556672 notifier 4221 bl.length()=714 2023-12-07T21:57:05.479 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.472+0000 7fcaf68cb700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-12-07T21:57:05.479 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.472+0000 7fcaf68cb700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-12-07T21:57:05.482 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.476+0000 7fcafaca1c80 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-12-07T21:57:05.482 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.476+0000 7fcafaca1c80 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-12-07T21:57:05.483 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.476+0000 7fcafaca1c80 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-12-07T21:57:05.483 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.480+0000 7fcaf60ca700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116995 cookie 94198478561456 notifier 4221 bl.length()=224 2023-12-07T21:57:05.483 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.480+0000 7fcaf60ca700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-12-07T21:57:05.483 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.480+0000 7fcaf60ca700 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-12-07T21:57:05.486 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.480+0000 7fcafaca1c80 10 cache put: name=default.rgw.meta+users.keys+JYGIAADVMWOHZERHZHDD info.flags=0x7 2023-12-07T21:57:05.487 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.480+0000 7fcafaca1c80 10 moving default.rgw.meta+users.keys+JYGIAADVMWOHZERHZHDD to cache LRU end 2023-12-07T21:57:05.487 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.480+0000 7fcafaca1c80 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.keys:JYGIAADVMWOHZERHZHDD, ofs0, ns] 2023-12-07T21:57:05.487 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.484+0000 7fcaf60ca700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116997 cookie 94198478541440 notifier 4221 bl.length()=201 2023-12-07T21:57:05.488 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.484+0000 7fcaf60ca700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+JYGIAADVMWOHZERHZHDD info.flags=0x7 2023-12-07T21:57:05.488 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.484+0000 7fcaf60ca700 10 rgw watcher librados: moving default.rgw.meta+users.keys+JYGIAADVMWOHZERHZHDD to cache LRU end 2023-12-07T21:57:05.488 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2023-12-07T21:57:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "user_id": "testx$tenanteduser.client.0", 2023-12-07T21:57:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2023-12-07T21:57:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2023-12-07T21:57:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "suspended": 0, 2023-12-07T21:57:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "max_buckets": 1000, 2023-12-07T21:57:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "subusers": [], 2023-12-07T21:57:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "keys": [ 2023-12-07T21:57:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-12-07T21:57:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "user": "testx$tenanteduser.client.0", 2023-12-07T21:57:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "access_key": "JYGIAADVMWOHZERHZHDD", 2023-12-07T21:57:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "secret_key": "jjrEygxpYD95OtwOh+ljxP2xvhewdwNwcMProRlJPwT1Za+DuAr4Jw==" 2023-12-07T21:57:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-12-07T21:57:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-12-07T21:57:05.489 INFO:teuthology.orchestra.run.smithi033.stdout: "swift_keys": [], 2023-12-07T21:57:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "caps": [ 2023-12-07T21:57:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: { 2023-12-07T21:57:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "user-policy", 2023-12-07T21:57:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "perm": "*" 2023-12-07T21:57:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: } 2023-12-07T21:57:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: ], 2023-12-07T21:57:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "op_mask": "read, write, delete", 2023-12-07T21:57:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "default_placement": "", 2023-12-07T21:57:05.490 INFO:teuthology.orchestra.run.smithi033.stdout: "default_storage_class": "", 2023-12-07T21:57:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "placement_tags": [], 2023-12-07T21:57:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "bucket_quota": { 2023-12-07T21:57:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": false, 2023-12-07T21:57:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "check_on_raw": false, 2023-12-07T21:57:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "max_size": -1, 2023-12-07T21:57:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "max_size_kb": 0, 2023-12-07T21:57:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "max_objects": -1 2023-12-07T21:57:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-12-07T21:57:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "user_quota": { 2023-12-07T21:57:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "enabled": false, 2023-12-07T21:57:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "check_on_raw": false, 2023-12-07T21:57:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "max_size": -1, 2023-12-07T21:57:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "max_size_kb": 0, 2023-12-07T21:57:05.491 INFO:teuthology.orchestra.run.smithi033.stdout: "max_objects": -1 2023-12-07T21:57:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: }, 2023-12-07T21:57:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "temp_url_keys": [], 2023-12-07T21:57:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "type": "rgw", 2023-12-07T21:57:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: "mfa_ids": [] 2023-12-07T21:57:05.492 INFO:teuthology.orchestra.run.smithi033.stdout:} 2023-12-07T21:57:05.492 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:57:05.493 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.488+0000 7fcafaca1c80 20 remove_watcher() i=0 2023-12-07T21:57:05.493 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.488+0000 7fcafaca1c80 2 removed watcher, disabling cache 2023-12-07T21:57:05.494 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.488+0000 7fcafaca1c80 20 remove_watcher() i=1 2023-12-07T21:57:05.496 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.492+0000 7fcafaca1c80 20 remove_watcher() i=2 2023-12-07T21:57:05.498 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.492+0000 7fcafaca1c80 20 remove_watcher() i=3 2023-12-07T21:57:05.500 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.496+0000 7fcafaca1c80 20 remove_watcher() i=4 2023-12-07T21:57:05.502 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.496+0000 7fcafaca1c80 20 remove_watcher() i=5 2023-12-07T21:57:05.504 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.500+0000 7fcafaca1c80 20 remove_watcher() i=6 2023-12-07T21:57:05.506 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.500+0000 7fcafaca1c80 20 remove_watcher() i=7 2023-12-07T21:57:05.506 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.500+0000 7fcaca7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T21:57:05.516 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 2221285457 --totp-seed SSUDSJI4URHIES4CKYUCKJHECU3R3J4AR4FAFK23LBG6OHKIBK5QPQQJ6Z26S726 --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-12-07T21:57:05.563 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setuser ceph since I am not root 2023-12-07T21:57:05.563 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setgroup ceph since I am not root 2023-12-07T21:57:05.576 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.572+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.577 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.572+0000 7f8ff70cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.577 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.572+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.577 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.572+0000 7f8ff70cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.577 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.572+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.578 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.572+0000 7f8ff70cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.578 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.572+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.578 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.572+0000 7f8ff70cdc80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.599 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.592+0000 7f8ff70cdc80 20 RGWRados::pool_iterate: got zonegroup_info.8b14f025-a9cb-4535-8f2a-c4795b716a58 2023-12-07T21:57:05.599 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.592+0000 7f8ff70cdc80 20 RGWRados::pool_iterate: got zone_info.5de1c49b-6b66-4f92-9f67-9eb1162319b3 2023-12-07T21:57:05.600 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.592+0000 7f8ff70cdc80 20 RGWRados::pool_iterate: got zone_names.default 2023-12-07T21:57:05.600 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.592+0000 7f8ff70cdc80 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-07T21:57:05.600 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.592+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.600 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.596+0000 7f8ff70cdc80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.600 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.596+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.600 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.596+0000 7f8ff70cdc80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T21:57:05.601 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.596+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.601 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.596+0000 7f8ff70cdc80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.601 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.596+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.602 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.596+0000 7f8ff70cdc80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T21:57:05.602 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.596+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.602 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.596+0000 7f8ff70cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.602 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.596+0000 7f8ff70cdc80 10 cannot find current period zonegroup using local zonegroup 2023-12-07T21:57:05.602 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.596+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.603 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.596+0000 7f8ff70cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.603 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.596+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.603 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.600+0000 7f8ff70cdc80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.604 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.600+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.604 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.600+0000 7f8ff70cdc80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T21:57:05.604 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.600+0000 7f8ff70cdc80 20 zonegroup default 2023-12-07T21:57:05.604 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.600+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.605 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.600+0000 7f8ff70cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.605 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.600+0000 7f8ff70cdc80 10 Cannot find current period zone using local zone 2023-12-07T21:57:05.605 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.600+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.605 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.600+0000 7f8ff70cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.605 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.600+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.606 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.600+0000 7f8ff70cdc80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T21:57:05.606 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.600+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.607 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.600+0000 7f8ff70cdc80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T21:57:05.607 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.600+0000 7f8ff70cdc80 20 zone default found 2023-12-07T21:57:05.607 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.600+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.608 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.604+0000 7f8ff70cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.608 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.604+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.608 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.604+0000 7f8ff70cdc80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T21:57:05.608 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.604+0000 7f8ff70cdc80 20 started sync module instance, tier type = 2023-12-07T21:57:05.608 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.604+0000 7f8ff70cdc80 20 started zone id=5de1c49b-6b66-4f92-9f67-9eb1162319b3 (name=default) with tier type = 2023-12-07T21:57:05.628 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.624+0000 7f8ff70cdc80 20 add_watcher() i=0 2023-12-07T21:57:05.628 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.624+0000 7f8ff70cdc80 20 add_watcher() i=1 2023-12-07T21:57:05.628 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.624+0000 7f8ff70cdc80 20 add_watcher() i=2 2023-12-07T21:57:05.629 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.624+0000 7f8ff70cdc80 20 add_watcher() i=3 2023-12-07T21:57:05.629 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.624+0000 7f8ff70cdc80 20 add_watcher() i=4 2023-12-07T21:57:05.629 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.624+0000 7f8ff70cdc80 20 add_watcher() i=5 2023-12-07T21:57:05.629 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.624+0000 7f8ff70cdc80 20 add_watcher() i=6 2023-12-07T21:57:05.629 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.624+0000 7f8ff70cdc80 20 add_watcher() i=7 2023-12-07T21:57:05.629 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.624+0000 7f8ff70cdc80 2 all 8 watchers are set, enabling cache 2023-12-07T21:57:05.632 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.628+0000 7f8fc67e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T21:57:05.632 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.628+0000 7f8ff70cdc80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-12-07T21:57:05.632 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.628+0000 7f8ff70cdc80 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-07T21:57:05.639 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.632+0000 7f8ff70cdc80 5 note: GC not initialized 2023-12-07T21:57:05.641 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.636+0000 7f8f357fa700 20 reqs_thread_entry: start 2023-12-07T21:57:05.641 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.636+0000 7f8f2effd700 20 reqs_thread_entry: start 2023-12-07T21:57:05.641 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.636+0000 7f8ff70cdc80 20 init_complete bucket index max shards: 11 2023-12-07T21:57:05.642 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.636+0000 7f8ff70cdc80 10 rgw notify: Started notification manager with: 1 workers 2023-12-07T21:57:05.642 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.636+0000 7f8f2d7fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Dec 7 21:57:36 2023 2023-12-07T21:57:05.642 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T21:57:05.642 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.636+0000 7f8f2cff9700 20 reqs_thread_entry: start 2023-12-07T21:57:05.642 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.636+0000 7f8ff70cdc80 20 get_system_obj_state: rctx=0x7ffe80e48648 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55eff9175020 s->prefetch_data=0 2023-12-07T21:57:05.642 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.636+0000 7f8ff70cdc80 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-12-07T21:57:05.643 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.636+0000 7f8ff70cdc80 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-12-07T21:57:05.643 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.636+0000 7f8ff70cdc80 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-12-07T21:57:05.643 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.636+0000 7f8ff70cdc80 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-07T21:57:05.644 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.636+0000 7f8ff70cdc80 20 get_system_obj_state: s->obj_tag was set empty 2023-12-07T21:57:05.644 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.636+0000 7f8ff70cdc80 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x11, cached=0x16) 2023-12-07T21:57:05.644 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.636+0000 7f8ff70cdc80 20 rados->read ofs=0 len=0 2023-12-07T21:57:05.645 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.640+0000 7f8ff70cdc80 20 rados_obj.operate() r=0 bl.length=514 2023-12-07T21:57:05.645 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.640+0000 7f8ff70cdc80 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x11 2023-12-07T21:57:05.645 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.640+0000 7f8ff70cdc80 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-12-07T21:57:05.648 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.644+0000 7f8ff70cdc80 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-12-07T21:57:05.648 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.644+0000 7f8ff70cdc80 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-12-07T21:57:05.649 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.644+0000 7f8ff70cdc80 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-12-07T21:57:05.649 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.644+0000 7f8feeffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 115964116996 cookie 94489165950320 notifier 4227 bl.length()=728 2023-12-07T21:57:05.649 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.644+0000 7f8feeffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-12-07T21:57:05.649 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.644+0000 7f8feeffd700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-12-07T21:57:05.654 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.648+0000 7f8ff70cdc80 20 remove_watcher() i=0 2023-12-07T21:57:05.654 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.648+0000 7f8ff70cdc80 2 removed watcher, disabling cache 2023-12-07T21:57:05.656 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.652+0000 7f8ff70cdc80 20 remove_watcher() i=1 2023-12-07T21:57:05.657 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.652+0000 7f8ff70cdc80 20 remove_watcher() i=2 2023-12-07T21:57:05.660 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.656+0000 7f8ff70cdc80 20 remove_watcher() i=3 2023-12-07T21:57:05.662 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.656+0000 7f8ff70cdc80 20 remove_watcher() i=4 2023-12-07T21:57:05.664 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.660+0000 7f8ff70cdc80 20 remove_watcher() i=5 2023-12-07T21:57:05.666 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.660+0000 7f8ff70cdc80 20 remove_watcher() i=6 2023-12-07T21:57:05.668 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.664+0000 7f8ff70cdc80 20 remove_watcher() i=7 2023-12-07T21:57:05.669 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T21:57:05.664+0000 7f8fc67e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T21:57:05.679 INFO:tasks.s3tests:Configuring s3-tests... 2023-12-07T21:57:05.679 DEBUG:teuthology.orchestra.run.smithi033:> cd /home/ubuntu/cephtest/s3-tests && ./bootstrap 2023-12-07T21:57:05.860 INFO:teuthology.orchestra.run.smithi033.stderr:./bootstrap: missing required DEB packages. Installing via sudo. 2023-12-07T21:57:05.929 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-07T21:57:06.145 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-12-07T21:57:06.146 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-12-07T21:57:06.353 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-12-07T21:57:06.354 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-07T21:57:06.354 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-07T21:57:06.355 INFO:teuthology.orchestra.run.smithi033.stdout:The following additional packages will be installed: 2023-12-07T21:57:06.356 INFO:teuthology.orchestra.run.smithi033.stdout: libevent-extra-2.1-7 libevent-openssl-2.1-7 libxml2 libxslt1.1 2023-12-07T21:57:06.357 INFO:teuthology.orchestra.run.smithi033.stdout:Suggested packages: 2023-12-07T21:57:06.357 INFO:teuthology.orchestra.run.smithi033.stdout: pkg-config 2023-12-07T21:57:06.397 INFO:teuthology.orchestra.run.smithi033.stdout:The following NEW packages will be installed: 2023-12-07T21:57:06.397 INFO:teuthology.orchestra.run.smithi033.stdout: libevent-dev libevent-extra-2.1-7 libevent-openssl-2.1-7 libxml2-dev 2023-12-07T21:57:06.398 INFO:teuthology.orchestra.run.smithi033.stdout: libxslt1-dev 2023-12-07T21:57:06.400 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages will be upgraded: 2023-12-07T21:57:06.400 INFO:teuthology.orchestra.run.smithi033.stdout: libxml2 libxslt1.1 2023-12-07T21:57:06.464 INFO:teuthology.orchestra.run.smithi033.stdout:2 upgraded, 5 newly installed, 0 to remove and 255 not upgraded. 2023-12-07T21:57:06.464 INFO:teuthology.orchestra.run.smithi033.stdout:Need to get 2,082 kB of archives. 2023-12-07T21:57:06.464 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 7,578 kB of additional disk space will be used. 2023-12-07T21:57:06.464 INFO:teuthology.orchestra.run.smithi033.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libxml2 amd64 2.9.10+dfsg-5ubuntu0.20.04.6 [640 kB] 2023-12-07T21:57:06.643 INFO:teuthology.orchestra.run.smithi033.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libevent-extra-2.1-7 amd64 2.1.11-stable-1 [60.0 kB] 2023-12-07T21:57:06.648 INFO:teuthology.orchestra.run.smithi033.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libevent-openssl-2.1-7 amd64 2.1.11-stable-1 [14.3 kB] 2023-12-07T21:57:06.648 INFO:teuthology.orchestra.run.smithi033.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libevent-dev amd64 2.1.11-stable-1 [261 kB] 2023-12-07T21:57:06.653 INFO:teuthology.orchestra.run.smithi033.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libxml2-dev amd64 2.9.10+dfsg-5ubuntu0.20.04.6 [735 kB] 2023-12-07T21:57:06.678 INFO:teuthology.orchestra.run.smithi033.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libxslt1.1 amd64 1.1.34-4ubuntu0.20.04.1 [151 kB] 2023-12-07T21:57:06.681 INFO:teuthology.orchestra.run.smithi033.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libxslt1-dev amd64 1.1.34-4ubuntu0.20.04.1 [219 kB] 2023-12-07T21:57:06.959 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: unable to initialize frontend: Dialog 2023-12-07T21:57:06.959 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2023-12-07T21:57:06.959 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: falling back to frontend: Readline 2023-12-07T21:57:06.987 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: unable to initialize frontend: Readline 2023-12-07T21:57:06.987 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: (This frontend requires a controlling tty.) 2023-12-07T21:57:06.987 INFO:teuthology.orchestra.run.smithi033.stderr:debconf: falling back to frontend: Teletype 2023-12-07T21:57:06.993 INFO:teuthology.orchestra.run.smithi033.stderr:dpkg-preconfigure: unable to re-open stdin: 2023-12-07T21:57:07.050 INFO:teuthology.orchestra.run.smithi033.stdout:Fetched 2,082 kB in 0s (7,528 kB/s) 2023-12-07T21:57:07.273 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 149337 files and directories currently installed.) 2023-12-07T21:57:07.278 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../0-libxml2_2.9.10+dfsg-5ubuntu0.20.04.6_amd64.deb ... 2023-12-07T21:57:07.462 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libxml2:amd64 (2.9.10+dfsg-5ubuntu0.20.04.6) over (2.9.10+dfsg-5ubuntu0.20.04.4) ... 2023-12-07T21:57:07.934 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libevent-extra-2.1-7:amd64. 2023-12-07T21:57:07.951 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../1-libevent-extra-2.1-7_2.1.11-stable-1_amd64.deb ... 2023-12-07T21:57:07.992 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libevent-extra-2.1-7:amd64 (2.1.11-stable-1) ... 2023-12-07T21:57:08.370 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libevent-openssl-2.1-7:amd64. 2023-12-07T21:57:08.387 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../2-libevent-openssl-2.1-7_2.1.11-stable-1_amd64.deb ... 2023-12-07T21:57:08.428 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libevent-openssl-2.1-7:amd64 (2.1.11-stable-1) ... 2023-12-07T21:57:08.710 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libevent-dev. 2023-12-07T21:57:08.723 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../3-libevent-dev_2.1.11-stable-1_amd64.deb ... 2023-12-07T21:57:08.765 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libevent-dev (2.1.11-stable-1) ... 2023-12-07T21:57:09.135 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libxml2-dev:amd64. 2023-12-07T21:57:09.152 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../4-libxml2-dev_2.9.10+dfsg-5ubuntu0.20.04.6_amd64.deb ... 2023-12-07T21:57:09.193 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libxml2-dev:amd64 (2.9.10+dfsg-5ubuntu0.20.04.6) ... 2023-12-07T21:57:09.718 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../5-libxslt1.1_1.1.34-4ubuntu0.20.04.1_amd64.deb ... 2023-12-07T21:57:09.865 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4ubuntu0.20.04.1) over (1.1.34-4) ... 2023-12-07T21:57:10.219 INFO:teuthology.orchestra.run.smithi033.stdout:Selecting previously unselected package libxslt1-dev:amd64. 2023-12-07T21:57:10.238 INFO:teuthology.orchestra.run.smithi033.stdout:Preparing to unpack .../6-libxslt1-dev_1.1.34-4ubuntu0.20.04.1_amd64.deb ... 2023-12-07T21:57:10.277 INFO:teuthology.orchestra.run.smithi033.stdout:Unpacking libxslt1-dev:amd64 (1.1.34-4ubuntu0.20.04.1) ... 2023-12-07T21:57:10.636 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libevent-extra-2.1-7:amd64 (2.1.11-stable-1) ... 2023-12-07T21:57:10.767 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libevent-openssl-2.1-7:amd64 (2.1.11-stable-1) ... 2023-12-07T21:57:10.893 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libevent-dev (2.1.11-stable-1) ... 2023-12-07T21:57:11.052 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libxml2:amd64 (2.9.10+dfsg-5ubuntu0.20.04.6) ... 2023-12-07T21:57:11.362 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libxml2-dev:amd64 (2.9.10+dfsg-5ubuntu0.20.04.6) ... 2023-12-07T21:57:11.521 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libxslt1.1:amd64 (1.1.34-4ubuntu0.20.04.1) ... 2023-12-07T21:57:11.639 INFO:teuthology.orchestra.run.smithi033.stdout:Setting up libxslt1-dev:amd64 (1.1.34-4ubuntu0.20.04.1) ... 2023-12-07T21:57:11.758 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-12-07T21:57:13.410 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-12-07T21:57:14.367 INFO:teuthology.orchestra.run.smithi033.stdout:created virtual environment CPython3.8.10.final.0-64 in 253ms 2023-12-07T21:57:14.367 INFO:teuthology.orchestra.run.smithi033.stdout: creator CPython3Posix(dest=/home/ubuntu/cephtest/s3-tests/virtualenv, clear=False, global=False) 2023-12-07T21:57:14.367 INFO:teuthology.orchestra.run.smithi033.stdout: seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, pkg_resources=latest, via=copy, app_data_dir=/home/ubuntu/.local/share/virtualenv/seed-app-data/v1.0.1.debian.1) 2023-12-07T21:57:14.368 INFO:teuthology.orchestra.run.smithi033.stdout: activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator 2023-12-07T21:57:15.670 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting pip 2023-12-07T21:57:15.677 INFO:teuthology.orchestra.run.smithi033.stdout: Using cached pip-23.3.1-py3-none-any.whl (2.1 MB) 2023-12-07T21:57:15.786 INFO:teuthology.orchestra.run.smithi033.stdout:Installing collected packages: pip 2023-12-07T21:57:15.786 INFO:teuthology.orchestra.run.smithi033.stdout: Attempting uninstall: pip 2023-12-07T21:57:15.786 INFO:teuthology.orchestra.run.smithi033.stdout: Found existing installation: pip 20.0.2 2023-12-07T21:57:15.840 INFO:teuthology.orchestra.run.smithi033.stdout: Uninstalling pip-20.0.2: 2023-12-07T21:57:15.844 INFO:teuthology.orchestra.run.smithi033.stdout: Successfully uninstalled pip-20.0.2 2023-12-07T21:57:16.790 INFO:teuthology.orchestra.run.smithi033.stdout:Successfully installed pip-23.3.1 2023-12-07T21:57:17.515 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting setuptools==44.1.0 2023-12-07T21:57:17.598 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB) 2023-12-07T21:57:17.739 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 583.4/583.4 kB 4.3 MB/s eta 0:00:00 2023-12-07T21:57:17.758 INFO:teuthology.orchestra.run.smithi033.stdout:Installing collected packages: setuptools 2023-12-07T21:57:17.759 INFO:teuthology.orchestra.run.smithi033.stdout: Attempting uninstall: setuptools 2023-12-07T21:57:17.759 INFO:teuthology.orchestra.run.smithi033.stdout: Found existing installation: setuptools 44.0.0 2023-12-07T21:57:17.768 INFO:teuthology.orchestra.run.smithi033.stdout: Uninstalling setuptools-44.0.0: 2023-12-07T21:57:17.775 INFO:teuthology.orchestra.run.smithi033.stdout: Successfully uninstalled setuptools-44.0.0 2023-12-07T21:57:18.028 INFO:teuthology.orchestra.run.smithi033.stdout:Successfully installed setuptools-44.1.0 2023-12-07T21:57:18.832 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting PyYAML (from -r requirements.txt (line 1)) 2023-12-07T21:57:18.896 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading PyYAML-6.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB) 2023-12-07T21:57:19.077 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting nose>=1.0.0 (from -r requirements.txt (line 2)) 2023-12-07T21:57:19.095 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading nose-1.3.7-py3-none-any.whl (154 kB) 2023-12-07T21:57:19.188 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 154.7/154.7 kB 1.7 MB/s eta 0:00:00 2023-12-07T21:57:19.298 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting boto>=2.6.0 (from -r requirements.txt (line 3)) 2023-12-07T21:57:19.317 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2023-12-07T21:57:19.430 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 1.4/1.4 MB 12.5 MB/s eta 0:00:00 2023-12-07T21:57:20.015 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting boto3>=1.0.0 (from -r requirements.txt (line 4)) 2023-12-07T21:57:20.033 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.33.10-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:20.730 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting botocore<1.28.0 (from -r requirements.txt (line 6)) 2023-12-07T21:57:20.748 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading botocore-1.27.96-py3-none-any.whl (9.3 MB) 2023-12-07T21:57:20.944 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 9.3/9.3 MB 48.7 MB/s eta 0:00:00 2023-12-07T21:57:21.087 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting munch>=2.0.0 (from -r requirements.txt (line 7)) 2023-12-07T21:57:21.116 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading munch-4.0.0-py2.py3-none-any.whl.metadata (5.9 kB) 2023-12-07T21:57:21.353 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting gevent>=1.0 (from -r requirements.txt (line 9)) 2023-12-07T21:57:21.371 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading gevent-23.9.1-cp38-cp38-manylinux_2_28_x86_64.whl.metadata (13 kB) 2023-12-07T21:57:21.538 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting isodate>=0.4.4 (from -r requirements.txt (line 10)) 2023-12-07T21:57:21.562 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB) 2023-12-07T21:57:21.620 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 41.7/41.7 kB 594.5 kB/s eta 0:00:00 2023-12-07T21:57:21.712 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting requests>=2.23.0 (from -r requirements.txt (line 11)) 2023-12-07T21:57:21.713 INFO:teuthology.orchestra.run.smithi033.stdout: Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB) 2023-12-07T21:57:21.821 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting pytz>=2011k (from -r requirements.txt (line 12)) 2023-12-07T21:57:21.822 INFO:teuthology.orchestra.run.smithi033.stdout: Using cached pytz-2023.3.post1-py2.py3-none-any.whl.metadata (22 kB) 2023-12-07T21:57:22.250 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting httplib2 (from -r requirements.txt (line 13)) 2023-12-07T21:57:22.269 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading httplib2-0.22.0-py3-none-any.whl (96 kB) 2023-12-07T21:57:22.330 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 96.9/96.9 kB 1.5 MB/s eta 0:00:00 2023-12-07T21:57:22.570 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting lxml (from -r requirements.txt (line 14)) 2023-12-07T21:57:22.588 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading lxml-4.9.3-cp38-cp38-manylinux_2_28_x86_64.whl.metadata (3.8 kB) 2023-12-07T21:57:22.799 INFO:teuthology.orchestra.run.smithi033.stdout:INFO: pip is looking at multiple versions of boto3 to determine which version is compatible with other requirements. This could take a while. 2023-12-07T21:57:22.803 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting boto3>=1.0.0 (from -r requirements.txt (line 4)) 2023-12-07T21:57:22.822 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.33.9-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:23.009 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.33.8-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:23.239 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.33.7-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:23.461 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.33.6-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:23.696 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.33.5-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:23.922 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.33.4-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:24.107 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.33.3-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:24.265 INFO:teuthology.orchestra.run.smithi033.stdout:INFO: pip is still looking at multiple versions of boto3 to determine which version is compatible with other requirements. This could take a while. 2023-12-07T21:57:24.306 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.33.2-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:24.553 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.33.1-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:24.771 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.33.0-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:25.005 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.29.7-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:25.193 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.29.6-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:25.439 INFO:teuthology.orchestra.run.smithi033.stdout:INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C. 2023-12-07T21:57:25.462 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.29.5-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:25.654 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.29.4-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:25.845 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.29.3-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:26.106 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.29.2-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:26.379 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.29.1-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:26.671 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.29.0-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:26.929 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.85-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:27.191 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.84-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:27.465 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.83-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:27.728 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.82-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:27.967 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.81-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:28.238 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.80-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:28.412 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.79-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:28.664 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.78-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:28.916 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.77-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:29.179 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.76-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:29.434 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.75-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:29.684 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.74-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:29.979 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.73-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:30.234 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.72-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:30.494 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.71-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:30.675 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.70-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:30.932 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.69-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:31.225 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.68-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:31.474 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.67-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:31.730 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.66-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:31.983 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.65-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:32.246 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.64-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:32.503 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.63-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:32.760 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.62-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:33.026 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.61-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:33.315 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.60-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:33.589 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.59-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:33.845 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.58-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:34.132 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.57-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:34.388 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.56-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:34.648 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.55-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:34.905 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.54-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:35.095 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.53-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:35.303 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.52-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:35.557 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.51-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:35.825 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.50-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:36.079 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.49-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:36.360 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.48-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:36.591 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.47-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:36.793 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.46-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:37.081 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.45-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:37.347 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.44-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:37.606 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.43-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:37.829 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.42-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:37.998 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.41-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:38.204 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.40-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:38.389 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.39-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:38.567 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.38-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:38.820 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.37-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:39.081 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.36-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:39.341 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.35-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:39.536 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.34-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:39.798 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.33-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:40.051 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.32-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:40.304 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.31-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:40.486 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.30-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:40.739 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.29-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:40.919 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.28-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:41.199 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.27-py3-none-any.whl.metadata (6.7 kB) 2023-12-07T21:57:41.450 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.26-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:41.632 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.25-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:41.883 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.24-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:42.069 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.23-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:42.241 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.22-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:42.504 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.21-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:42.764 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.20-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:43.034 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.19-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:43.308 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.18-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:43.486 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.17-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:43.866 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.16-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.15-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:44.400 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.14-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:44.656 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.13-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:44.870 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.12-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:45.059 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.11-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:45.269 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.10-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:45.527 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.9-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:45.803 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.8-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:45.983 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.7-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:46.245 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.6-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:46.505 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.5-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:46.761 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.4-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:47.055 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.3-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:47.341 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.2-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:47.594 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.1-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:48.314 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.28.0-py3-none-any.whl.metadata (6.6 kB) 2023-12-07T21:57:48.767 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.27.1-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:49.046 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.27.0-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:49.309 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.165-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:49.547 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.164-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:49.802 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.163-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:50.078 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.162-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:50.339 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.161-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:50.519 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.160-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:50.779 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.159-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:50.945 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.158-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:51.122 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.157-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:51.334 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.156-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:51.515 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.155-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:51.767 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.154-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:52.037 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.153-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:52.285 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.152-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:52.504 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.151-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:52.755 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.150-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:53.014 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.149-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:53.273 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.148-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:53.694 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.147-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:53.915 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.146-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:54.191 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.145-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:54.440 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.144-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:54.615 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.143-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:54.875 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.142-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:55.107 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.141-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:55.405 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.140-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:55.660 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.139-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:55.844 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.138-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:56.102 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.137-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:56.287 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.136-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:56.473 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.135-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:56.658 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.134-py3-none-any.whl.metadata (7.0 kB) 2023-12-07T21:57:56.840 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.133-py3-none-any.whl (135 kB) 2023-12-07T21:57:56.907 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 1.9 MB/s eta 0:00:00 2023-12-07T21:57:57.132 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.132-py3-none-any.whl (135 kB) 2023-12-07T21:57:57.201 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.0 MB/s eta 0:00:00 2023-12-07T21:57:57.407 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.131-py3-none-any.whl (135 kB) 2023-12-07T21:57:57.477 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 1.9 MB/s eta 0:00:00 2023-12-07T21:57:57.685 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.130-py3-none-any.whl (135 kB) 2023-12-07T21:57:57.752 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.0 MB/s eta 0:00:00 2023-12-07T21:57:57.954 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.129-py3-none-any.whl (135 kB) 2023-12-07T21:57:58.012 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:57:58.146 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.128-py3-none-any.whl (135 kB) 2023-12-07T21:57:58.212 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.0 MB/s eta 0:00:00 2023-12-07T21:57:58.346 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.127-py3-none-any.whl (135 kB) 2023-12-07T21:57:58.404 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.3 MB/s eta 0:00:00 2023-12-07T21:57:58.541 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.126-py3-none-any.whl (135 kB) 2023-12-07T21:57:58.597 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.5 MB/s eta 0:00:00 2023-12-07T21:57:58.730 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.125-py3-none-any.whl (135 kB) 2023-12-07T21:57:58.789 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.3 MB/s eta 0:00:00 2023-12-07T21:57:59.003 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.124-py3-none-any.whl (135 kB) 2023-12-07T21:57:59.064 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:57:59.276 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.123-py3-none-any.whl (135 kB) 2023-12-07T21:57:59.368 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 1.4 MB/s eta 0:00:00 2023-12-07T21:57:59.505 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.122-py3-none-any.whl (135 kB) 2023-12-07T21:57:59.566 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:57:59.779 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.121-py3-none-any.whl (135 kB) 2023-12-07T21:57:59.842 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:57:59.976 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.120-py3-none-any.whl (135 kB) 2023-12-07T21:58:00.034 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:00.250 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.119-py3-none-any.whl (135 kB) 2023-12-07T21:58:00.310 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.3 MB/s eta 0:00:00 2023-12-07T21:58:00.442 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.118-py3-none-any.whl (135 kB) 2023-12-07T21:58:00.510 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.0 MB/s eta 0:00:00 2023-12-07T21:58:00.713 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.117-py3-none-any.whl (135 kB) 2023-12-07T21:58:00.786 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 1.8 MB/s eta 0:00:00 2023-12-07T21:58:00.948 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.116-py3-none-any.whl (135 kB) 2023-12-07T21:58:01.045 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 1.4 MB/s eta 0:00:00 2023-12-07T21:58:01.247 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.115-py3-none-any.whl (135 kB) 2023-12-07T21:58:01.304 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:01.505 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.114-py3-none-any.whl (135 kB) 2023-12-07T21:58:01.563 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:01.764 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.113-py3-none-any.whl (135 kB) 2023-12-07T21:58:01.822 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:02.023 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.112-py3-none-any.whl (135 kB) 2023-12-07T21:58:02.081 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:02.282 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.111-py3-none-any.whl (135 kB) 2023-12-07T21:58:02.340 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:02.469 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.110-py3-none-any.whl (135 kB) 2023-12-07T21:58:02.524 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.5 MB/s eta 0:00:00 2023-12-07T21:58:02.724 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.109-py3-none-any.whl (135 kB) 2023-12-07T21:58:02.783 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.3 MB/s eta 0:00:00 2023-12-07T21:58:02.988 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.108-py3-none-any.whl (135 kB) 2023-12-07T21:58:03.049 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:03.179 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.107-py3-none-any.whl (135 kB) 2023-12-07T21:58:03.242 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:03.444 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.106-py3-none-any.whl (135 kB) 2023-12-07T21:58:03.535 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 1.5 MB/s eta 0:00:00 2023-12-07T21:58:03.740 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.105-py3-none-any.whl (135 kB) 2023-12-07T21:58:03.810 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 1.9 MB/s eta 0:00:00 2023-12-07T21:58:04.022 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.104-py3-none-any.whl (135 kB) 2023-12-07T21:58:04.086 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:04.305 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.103-py3-none-any.whl (135 kB) 2023-12-07T21:58:04.395 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.6/135.6 kB 1.5 MB/s eta 0:00:00 2023-12-07T21:58:04.599 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.102-py3-none-any.whl (135 kB) 2023-12-07T21:58:04.771 INFO:teuthology.orchestra.run.smithi033.stdout: ------------------------------------- 135.6/135.6 kB 751.2 kB/s eta 0:00:00 2023-12-07T21:58:04.907 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.101-py3-none-any.whl (135 kB) 2023-12-07T21:58:04.963 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.5/135.5 kB 2.5 MB/s eta 0:00:00 2023-12-07T21:58:05.167 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.100-py3-none-any.whl (135 kB) 2023-12-07T21:58:05.263 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.5/135.5 kB 1.4 MB/s eta 0:00:00 2023-12-07T21:58:05.469 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.99-py3-none-any.whl (135 kB) 2023-12-07T21:58:05.531 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.5/135.5 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:05.732 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.98-py3-none-any.whl (135 kB) 2023-12-07T21:58:05.798 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.5/135.5 kB 2.0 MB/s eta 0:00:00 2023-12-07T21:58:06.003 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.97-py3-none-any.whl (135 kB) 2023-12-07T21:58:06.090 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.5/135.5 kB 1.5 MB/s eta 0:00:00 2023-12-07T21:58:06.296 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.96-py3-none-any.whl (135 kB) 2023-12-07T21:58:06.358 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.5/135.5 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:06.561 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.95-py3-none-any.whl (135 kB) 2023-12-07T21:58:06.625 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.5/135.5 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:06.829 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.94-py3-none-any.whl (135 kB) 2023-12-07T21:58:06.918 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.1/135.1 kB 1.5 MB/s eta 0:00:00 2023-12-07T21:58:07.089 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.93-py3-none-any.whl (135 kB) 2023-12-07T21:58:07.151 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 135.1/135.1 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:07.280 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.92-py3-none-any.whl (134 kB) 2023-12-07T21:58:07.352 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 134.7/134.7 kB 1.9 MB/s eta 0:00:00 2023-12-07T21:58:07.552 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.91-py3-none-any.whl (134 kB) 2023-12-07T21:58:07.619 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 134.7/134.7 kB 2.0 MB/s eta 0:00:00 2023-12-07T21:58:07.747 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.90-py3-none-any.whl (134 kB) 2023-12-07T21:58:07.806 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 134.7/134.7 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:08.019 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.89-py3-none-any.whl (134 kB) 2023-12-07T21:58:08.079 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 134.7/134.7 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:08.282 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.88-py3-none-any.whl (134 kB) 2023-12-07T21:58:08.338 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 134.7/134.7 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:08.542 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.87-py3-none-any.whl (134 kB) 2023-12-07T21:58:08.604 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 134.7/134.7 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:08.808 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.86-py3-none-any.whl (134 kB) 2023-12-07T21:58:08.864 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 134.7/134.7 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:09.074 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.85-py3-none-any.whl (134 kB) 2023-12-07T21:58:09.132 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 134.7/134.7 kB 2.3 MB/s eta 0:00:00 2023-12-07T21:58:09.338 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.84-py3-none-any.whl (134 kB) 2023-12-07T21:58:09.416 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 134.7/134.7 kB 1.7 MB/s eta 0:00:00 2023-12-07T21:58:09.625 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.83-py3-none-any.whl (134 kB) 2023-12-07T21:58:09.683 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 134.7/134.7 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:09.888 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.82-py3-none-any.whl (134 kB) 2023-12-07T21:58:09.950 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 134.7/134.7 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:10.167 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.81-py3-none-any.whl (134 kB) 2023-12-07T21:58:10.251 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 134.7/134.7 kB 1.6 MB/s eta 0:00:00 2023-12-07T21:58:10.458 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.80-py3-none-any.whl (132 kB) 2023-12-07T21:58:10.518 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:10.644 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.79-py3-none-any.whl (132 kB) 2023-12-07T21:58:10.719 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 1.8 MB/s eta 0:00:00 2023-12-07T21:58:10.922 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.78-py3-none-any.whl (132 kB) 2023-12-07T21:58:10.977 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:11.180 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.77-py3-none-any.whl (132 kB) 2023-12-07T21:58:11.237 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:11.366 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.76-py3-none-any.whl (132 kB) 2023-12-07T21:58:11.421 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:11.552 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.75-py3-none-any.whl (132 kB) 2023-12-07T21:58:11.605 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.5 MB/s eta 0:00:00 2023-12-07T21:58:11.812 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.74-py3-none-any.whl (132 kB) 2023-12-07T21:58:11.897 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 1.5 MB/s eta 0:00:00 2023-12-07T21:58:12.103 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.73-py3-none-any.whl (132 kB) 2023-12-07T21:58:12.173 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 1.9 MB/s eta 0:00:00 2023-12-07T21:58:12.302 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.72-py3-none-any.whl (132 kB) 2023-12-07T21:58:12.365 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:12.495 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.71-py3-none-any.whl (132 kB) 2023-12-07T21:58:12.558 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:12.775 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.70-py3-none-any.whl (132 kB) 2023-12-07T21:58:12.832 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.3 MB/s eta 0:00:00 2023-12-07T21:58:13.073 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.69-py3-none-any.whl (132 kB) 2023-12-07T21:58:13.159 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 1.5 MB/s eta 0:00:00 2023-12-07T21:58:13.307 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.68-py3-none-any.whl (132 kB) 2023-12-07T21:58:13.376 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 1.9 MB/s eta 0:00:00 2023-12-07T21:58:13.584 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.67-py3-none-any.whl (132 kB) 2023-12-07T21:58:13.643 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.3 MB/s eta 0:00:00 2023-12-07T21:58:13.809 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.66-py3-none-any.whl (132 kB) 2023-12-07T21:58:13.877 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 1.9 MB/s eta 0:00:00 2023-12-07T21:58:14.051 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.65-py3-none-any.whl (132 kB) 2023-12-07T21:58:14.145 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 1.4 MB/s eta 0:00:00 2023-12-07T21:58:14.368 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.64-py3-none-any.whl (132 kB) 2023-12-07T21:58:14.429 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:14.653 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.63-py3-none-any.whl (132 kB) 2023-12-07T21:58:14.712 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:14.913 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.62-py3-none-any.whl (132 kB) 2023-12-07T21:58:14.971 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.3 MB/s eta 0:00:00 2023-12-07T21:58:15.121 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.61-py3-none-any.whl (132 kB) 2023-12-07T21:58:15.214 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 1.4 MB/s eta 0:00:00 2023-12-07T21:58:15.350 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.60-py3-none-any.whl (132 kB) 2023-12-07T21:58:15.423 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 1.8 MB/s eta 0:00:00 2023-12-07T21:58:15.629 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.59-py3-none-any.whl (132 kB) 2023-12-07T21:58:15.690 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:15.821 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.58-py3-none-any.whl (132 kB) 2023-12-07T21:58:15.874 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:16.036 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.57-py3-none-any.whl (132 kB) 2023-12-07T21:58:16.100 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:16.229 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.56-py3-none-any.whl (132 kB) 2023-12-07T21:58:16.292 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:16.421 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.55-py3-none-any.whl (132 kB) 2023-12-07T21:58:16.484 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:16.685 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.54-py3-none-any.whl (132 kB) 2023-12-07T21:58:16.752 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.0 MB/s eta 0:00:00 2023-12-07T21:58:16.954 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.53-py3-none-any.whl (132 kB) 2023-12-07T21:58:17.027 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 1.8 MB/s eta 0:00:00 2023-12-07T21:58:17.195 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.52-py3-none-any.whl (132 kB) 2023-12-07T21:58:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 1.4 MB/s eta 0:00:00 2023-12-07T21:58:17.505 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.51-py3-none-any.whl (132 kB) 2023-12-07T21:58:17.561 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.3 MB/s eta 0:00:00 2023-12-07T21:58:17.776 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.50-py3-none-any.whl (132 kB) 2023-12-07T21:58:17.837 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:18.043 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.49-py3-none-any.whl (132 kB) 2023-12-07T21:58:18.130 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 1.5 MB/s eta 0:00:00 2023-12-07T21:58:18.294 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.48-py3-none-any.whl (132 kB) 2023-12-07T21:58:18.347 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.5 MB/s eta 0:00:00 2023-12-07T21:58:18.548 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.47-py3-none-any.whl (132 kB) 2023-12-07T21:58:18.606 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.3 MB/s eta 0:00:00 2023-12-07T21:58:18.735 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.46-py3-none-any.whl (132 kB) 2023-12-07T21:58:18.790 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:19.008 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.45-py3-none-any.whl (132 kB) 2023-12-07T21:58:19.074 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.0 MB/s eta 0:00:00 2023-12-07T21:58:19.288 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.44-py3-none-any.whl (132 kB) 2023-12-07T21:58:19.368 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 1.6 MB/s eta 0:00:00 2023-12-07T21:58:19.600 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.43-py3-none-any.whl (132 kB) 2023-12-07T21:58:19.659 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:19.792 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.42-py3-none-any.whl (132 kB) 2023-12-07T21:58:19.851 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.3 MB/s eta 0:00:00 2023-12-07T21:58:20.080 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.41-py3-none-any.whl (132 kB) 2023-12-07T21:58:20.135 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:20.340 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.40-py3-none-any.whl (132 kB) 2023-12-07T21:58:20.402 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:20.630 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.39-py3-none-any.whl (132 kB) 2023-12-07T21:58:20.812 INFO:teuthology.orchestra.run.smithi033.stdout: ------------------------------------- 132.7/132.7 kB 691.0 kB/s eta 0:00:00 2023-12-07T21:58:21.043 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.38-py3-none-any.whl (132 kB) 2023-12-07T21:58:21.104 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:21.238 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.37-py3-none-any.whl (132 kB) 2023-12-07T21:58:21.288 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.7 MB/s eta 0:00:00 2023-12-07T21:58:21.418 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.36-py3-none-any.whl (132 kB) 2023-12-07T21:58:21.472 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.5 MB/s eta 0:00:00 2023-12-07T21:58:21.672 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.35-py3-none-any.whl (132 kB) 2023-12-07T21:58:21.739 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.0 MB/s eta 0:00:00 2023-12-07T21:58:21.903 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.34-py3-none-any.whl (132 kB) 2023-12-07T21:58:21.957 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.5 MB/s eta 0:00:00 2023-12-07T21:58:22.089 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.33-py3-none-any.whl (132 kB) 2023-12-07T21:58:22.165 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 1.7 MB/s eta 0:00:00 2023-12-07T21:58:22.295 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.32-py3-none-any.whl (132 kB) 2023-12-07T21:58:22.358 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.7/132.7 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:22.490 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.31-py3-none-any.whl (132 kB) 2023-12-07T21:58:22.559 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.6/132.6 kB 1.9 MB/s eta 0:00:00 2023-12-07T21:58:22.697 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.30-py3-none-any.whl (132 kB) 2023-12-07T21:58:22.759 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.6/132.6 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:22.976 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.29-py3-none-any.whl (132 kB) 2023-12-07T21:58:23.052 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.6/132.6 kB 1.7 MB/s eta 0:00:00 2023-12-07T21:58:23.189 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.28-py3-none-any.whl (132 kB) 2023-12-07T21:58:23.243 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.6/132.6 kB 2.5 MB/s eta 0:00:00 2023-12-07T21:58:23.446 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.27-py3-none-any.whl (132 kB) 2023-12-07T21:58:23.536 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.6/132.6 kB 1.4 MB/s eta 0:00:00 2023-12-07T21:58:23.738 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.26-py3-none-any.whl (132 kB) 2023-12-07T21:58:23.794 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.6/132.6 kB 2.3 MB/s eta 0:00:00 2023-12-07T21:58:24.030 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.25-py3-none-any.whl (132 kB) 2023-12-07T21:58:24.087 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.6/132.6 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:24.218 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.24-py3-none-any.whl (132 kB) 2023-12-07T21:58:24.279 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.6/132.6 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:24.483 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.23-py3-none-any.whl (132 kB) 2023-12-07T21:58:24.538 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.6/132.6 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:24.741 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.22-py3-none-any.whl (132 kB) 2023-12-07T21:58:24.814 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.6/132.6 kB 1.8 MB/s eta 0:00:00 2023-12-07T21:58:25.023 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.21-py3-none-any.whl (132 kB) 2023-12-07T21:58:25.082 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.6/132.6 kB 2.3 MB/s eta 0:00:00 2023-12-07T21:58:25.284 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.20-py3-none-any.whl (132 kB) 2023-12-07T21:58:25.332 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.6/132.6 kB 2.8 MB/s eta 0:00:00 2023-12-07T21:58:25.530 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.19-py3-none-any.whl (132 kB) 2023-12-07T21:58:25.633 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.6/132.6 kB 1.2 MB/s eta 0:00:00 2023-12-07T21:58:25.812 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.18-py3-none-any.whl (132 kB) 2023-12-07T21:58:25.875 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.6/132.6 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:26.081 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.17-py3-none-any.whl (132 kB) 2023-12-07T21:58:26.142 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.6/132.6 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:26.347 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.16-py3-none-any.whl (132 kB) 2023-12-07T21:58:26.423 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 1.7 MB/s eta 0:00:00 2023-12-07T21:58:26.623 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.15-py3-none-any.whl (132 kB) 2023-12-07T21:58:26.686 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:26.889 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.14-py3-none-any.whl (132 kB) 2023-12-07T21:58:26.952 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:27.170 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.13-py3-none-any.whl (132 kB) 2023-12-07T21:58:27.271 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 1.3 MB/s eta 0:00:00 2023-12-07T21:58:27.405 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.12-py3-none-any.whl (132 kB) 2023-12-07T21:58:27.462 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.3 MB/s eta 0:00:00 2023-12-07T21:58:27.665 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.11-py3-none-any.whl (132 kB) 2023-12-07T21:58:27.838 INFO:teuthology.orchestra.run.smithi033.stdout: ------------------------------------- 132.5/132.5 kB 726.3 kB/s eta 0:00:00 2023-12-07T21:58:28.044 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.10-py3-none-any.whl (132 kB) 2023-12-07T21:58:28.130 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 1.5 MB/s eta 0:00:00 2023-12-07T21:58:28.333 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.9-py3-none-any.whl (132 kB) 2023-12-07T21:58:28.388 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:28.552 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.8-py3-none-any.whl (132 kB) 2023-12-07T21:58:28.614 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:28.746 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.7-py3-none-any.whl (132 kB) 2023-12-07T21:58:28.807 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:29.017 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.6-py3-none-any.whl (132 kB) 2023-12-07T21:58:29.074 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.3 MB/s eta 0:00:00 2023-12-07T21:58:29.282 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.5-py3-none-any.whl (132 kB) 2023-12-07T21:58:29.367 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 1.5 MB/s eta 0:00:00 2023-12-07T21:58:29.571 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.4-py3-none-any.whl (132 kB) 2023-12-07T21:58:29.626 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:29.787 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.3-py3-none-any.whl (132 kB) 2023-12-07T21:58:29.851 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.0 MB/s eta 0:00:00 2023-12-07T21:58:29.982 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.2-py3-none-any.whl (132 kB) 2023-12-07T21:58:30.044 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:30.247 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.1-py3-none-any.whl (132 kB) 2023-12-07T21:58:30.311 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.0 MB/s eta 0:00:00 2023-12-07T21:58:30.517 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.26.0-py3-none-any.whl (132 kB) 2023-12-07T21:58:30.603 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 1.5 MB/s eta 0:00:00 2023-12-07T21:58:30.809 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.25.5-py3-none-any.whl (132 kB) 2023-12-07T21:58:30.862 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.5 MB/s eta 0:00:00 2023-12-07T21:58:31.070 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.25.4-py3-none-any.whl (132 kB) 2023-12-07T21:58:31.138 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.0 MB/s eta 0:00:00 2023-12-07T21:58:31.300 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.25.3-py3-none-any.whl (132 kB) 2023-12-07T21:58:31.355 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.4 MB/s eta 0:00:00 2023-12-07T21:58:31.488 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.25.2-py3-none-any.whl (132 kB) 2023-12-07T21:58:31.547 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.2 MB/s eta 0:00:00 2023-12-07T21:58:31.677 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.25.1-py3-none-any.whl (132 kB) 2023-12-07T21:58:31.732 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.5 MB/s eta 0:00:00 2023-12-07T21:58:31.870 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.25.0-py3-none-any.whl (132 kB) 2023-12-07T21:58:31.932 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.1 MB/s eta 0:00:00 2023-12-07T21:58:32.092 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading boto3-1.24.96-py3-none-any.whl (132 kB) 2023-12-07T21:58:32.158 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 132.5/132.5 kB 2.0 MB/s eta 0:00:00 2023-12-07T21:58:32.389 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting jmespath<2.0.0,>=0.7.1 (from boto3>=1.0.0->-r requirements.txt (line 4)) 2023-12-07T21:58:32.406 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading jmespath-1.0.1-py3-none-any.whl (20 kB) 2023-12-07T21:58:32.553 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting s3transfer<0.7.0,>=0.6.0 (from boto3>=1.0.0->-r requirements.txt (line 4)) 2023-12-07T21:58:32.571 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading s3transfer-0.6.2-py3-none-any.whl.metadata (1.8 kB) 2023-12-07T21:58:32.738 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.28.0->-r requirements.txt (line 6)) 2023-12-07T21:58:32.802 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-12-07T21:58:33.204 INFO:teuthology.orchestra.run.smithi033.stdout: ------------------------------------- 247.7/247.7 kB 791.5 kB/s eta 0:00:00 2023-12-07T21:58:33.320 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting urllib3<1.27,>=1.25.4 (from botocore<1.28.0->-r requirements.txt (line 6)) 2023-12-07T21:58:33.340 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading urllib3-1.26.18-py2.py3-none-any.whl.metadata (48 kB) 2023-12-07T21:58:33.405 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 48.9/48.9 kB 658.2 kB/s eta 0:00:00 2023-12-07T21:58:33.576 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting zope.event (from gevent>=1.0->-r requirements.txt (line 9)) 2023-12-07T21:58:33.599 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading zope.event-5.0-py3-none-any.whl.metadata (4.4 kB) 2023-12-07T21:58:33.845 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting zope.interface (from gevent>=1.0->-r requirements.txt (line 9)) 2023-12-07T21:58:33.862 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading zope.interface-6.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (41 kB) 2023-12-07T21:58:33.957 INFO:teuthology.orchestra.run.smithi033.stdout: --------------------------------------- 41.7/41.7 kB 351.6 kB/s eta 0:00:00 2023-12-07T21:58:34.156 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting greenlet>=2.0.0 (from gevent>=1.0->-r requirements.txt (line 9)) 2023-12-07T21:58:34.158 INFO:teuthology.orchestra.run.smithi033.stdout: Using cached greenlet-3.0.1-cp38-cp38-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (3.7 kB) 2023-12-07T21:58:34.222 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting six (from isodate>=0.4.4->-r requirements.txt (line 10)) 2023-12-07T21:58:34.223 INFO:teuthology.orchestra.run.smithi033.stdout: Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-12-07T21:58:34.380 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting charset-normalizer<4,>=2 (from requests>=2.23.0->-r requirements.txt (line 11)) 2023-12-07T21:58:34.381 INFO:teuthology.orchestra.run.smithi033.stdout: Using cached charset_normalizer-3.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (33 kB) 2023-12-07T21:58:34.449 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting idna<4,>=2.5 (from requests>=2.23.0->-r requirements.txt (line 11)) 2023-12-07T21:58:34.450 INFO:teuthology.orchestra.run.smithi033.stdout: Using cached idna-3.6-py3-none-any.whl.metadata (9.9 kB) 2023-12-07T21:58:34.533 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting certifi>=2017.4.17 (from requests>=2.23.0->-r requirements.txt (line 11)) 2023-12-07T21:58:34.534 INFO:teuthology.orchestra.run.smithi033.stdout: Using cached certifi-2023.11.17-py3-none-any.whl.metadata (2.2 kB) 2023-12-07T21:58:34.670 INFO:teuthology.orchestra.run.smithi033.stdout:Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 (from httplib2->-r requirements.txt (line 13)) 2023-12-07T21:58:34.708 INFO:teuthology.orchestra.run.smithi033.stdout: Downloading pyparsing-3.1.1-py3-none-any.whl.metadata (5.1 kB) 2023-12-07T21:58:35.073 INFO:teuthology.orchestra.run.smithi033.stdout:Requirement already satisfied: setuptools in ./virtualenv/lib/python3.8/site-packages (from zope.event->gevent>=1.0->-r requirements.txt (line 9)) (44.1.0) 2023-12-07T21:58:35.118 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading PyYAML-6.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (736 kB) 2023-12-07T21:58:35.219 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 736.6/736.6 kB 7.6 MB/s eta 0:00:00 2023-12-07T21:58:35.254 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading munch-4.0.0-py2.py3-none-any.whl (9.9 kB) 2023-12-07T21:58:35.361 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading gevent-23.9.1-cp38-cp38-manylinux_2_28_x86_64.whl (6.5 MB) 2023-12-07T21:58:35.636 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 6.5/6.5 MB 23.8 MB/s eta 0:00:00 2023-12-07T21:58:35.639 INFO:teuthology.orchestra.run.smithi033.stdout:Using cached requests-2.31.0-py3-none-any.whl (62 kB) 2023-12-07T21:58:35.642 INFO:teuthology.orchestra.run.smithi033.stdout:Using cached pytz-2023.3.post1-py2.py3-none-any.whl (502 kB) 2023-12-07T21:58:35.666 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading lxml-4.9.3-cp38-cp38-manylinux_2_28_x86_64.whl (8.0 MB) 2023-12-07T21:58:35.891 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 8.0/8.0 MB 36.5 MB/s eta 0:00:00 2023-12-07T21:58:35.894 INFO:teuthology.orchestra.run.smithi033.stdout:Using cached certifi-2023.11.17-py3-none-any.whl (162 kB) 2023-12-07T21:58:35.898 INFO:teuthology.orchestra.run.smithi033.stdout:Using cached charset_normalizer-3.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (141 kB) 2023-12-07T21:58:35.899 INFO:teuthology.orchestra.run.smithi033.stdout:Using cached greenlet-3.0.1-cp38-cp38-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (618 kB) 2023-12-07T21:58:35.902 INFO:teuthology.orchestra.run.smithi033.stdout:Using cached idna-3.6-py3-none-any.whl (61 kB) 2023-12-07T21:58:35.921 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB) 2023-12-07T21:58:36.004 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 103.1/103.1 kB 1.2 MB/s eta 0:00:00 2023-12-07T21:58:36.025 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading s3transfer-0.6.2-py3-none-any.whl (79 kB) 2023-12-07T21:58:36.089 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 79.8/79.8 kB 1.2 MB/s eta 0:00:00 2023-12-07T21:58:36.109 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2023-12-07T21:58:36.165 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 143.8/143.8 kB 2.6 MB/s eta 0:00:00 2023-12-07T21:58:36.184 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading zope.event-5.0-py3-none-any.whl (6.8 kB) 2023-12-07T21:58:36.260 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading zope.interface-6.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (249 kB) 2023-12-07T21:58:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: ---------------------------------------- 249.3/249.3 kB 4.1 MB/s eta 0:00:00 2023-12-07T21:58:36.513 INFO:teuthology.orchestra.run.smithi033.stdout:Installing collected packages: pytz, nose, boto, zope.interface, zope.event, urllib3, six, PyYAML, pyparsing, munch, lxml, jmespath, idna, greenlet, charset-normalizer, certifi, requests, python-dateutil, isodate, httplib2, gevent, botocore, s3transfer, boto3 2023-12-07T21:58:40.606 INFO:teuthology.orchestra.run.smithi033.stdout:Successfully installed PyYAML-6.0.1 boto-2.49.0 boto3-1.24.96 botocore-1.27.96 certifi-2023.11.17 charset-normalizer-3.3.2 gevent-23.9.1 greenlet-3.0.1 httplib2-0.22.0 idna-3.6 isodate-0.6.1 jmespath-1.0.1 lxml-4.9.3 munch-4.0.0 nose-1.3.7 pyparsing-3.1.1 python-dateutil-2.8.2 pytz-2023.3.post1 requests-2.31.0 s3transfer-0.6.2 six-1.16.0 urllib3-1.26.18 zope.event-5.0 zope.interface-6.1 2023-12-07T21:58:41.068 INFO:teuthology.orchestra.run.smithi033.stdout:running develop 2023-12-07T21:58:41.089 INFO:teuthology.orchestra.run.smithi033.stdout:running egg_info 2023-12-07T21:58:41.090 INFO:teuthology.orchestra.run.smithi033.stdout:creating s3tests.egg-info 2023-12-07T21:58:41.090 INFO:teuthology.orchestra.run.smithi033.stdout:writing s3tests.egg-info/PKG-INFO 2023-12-07T21:58:41.090 INFO:teuthology.orchestra.run.smithi033.stdout:writing dependency_links to s3tests.egg-info/dependency_links.txt 2023-12-07T21:58:41.090 INFO:teuthology.orchestra.run.smithi033.stdout:writing requirements to s3tests.egg-info/requires.txt 2023-12-07T21:58:41.090 INFO:teuthology.orchestra.run.smithi033.stdout:writing top-level names to s3tests.egg-info/top_level.txt 2023-12-07T21:58:41.090 INFO:teuthology.orchestra.run.smithi033.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-12-07T21:58:41.119 INFO:teuthology.orchestra.run.smithi033.stdout:reading manifest file 's3tests.egg-info/SOURCES.txt' 2023-12-07T21:58:41.120 INFO:teuthology.orchestra.run.smithi033.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-12-07T21:58:41.121 INFO:teuthology.orchestra.run.smithi033.stdout:running build_ext 2023-12-07T21:58:41.121 INFO:teuthology.orchestra.run.smithi033.stdout:Creating /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages/s3tests.egg-link (link to .) 2023-12-07T21:58:41.121 INFO:teuthology.orchestra.run.smithi033.stdout:Adding s3tests 0.0.1 to easy-install.pth file 2023-12-07T21:58:41.122 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.122 INFO:teuthology.orchestra.run.smithi033.stdout:Installed /home/ubuntu/cephtest/s3-tests 2023-12-07T21:58:41.122 INFO:teuthology.orchestra.run.smithi033.stdout:Processing dependencies for s3tests==0.0.1 2023-12-07T21:58:41.190 INFO:teuthology.orchestra.run.smithi033.stdout:Searching for isodate==0.6.1 2023-12-07T21:58:41.190 INFO:teuthology.orchestra.run.smithi033.stdout:Best match: isodate 0.6.1 2023-12-07T21:58:41.190 INFO:teuthology.orchestra.run.smithi033.stdout:Adding isodate 0.6.1 to easy-install.pth file 2023-12-07T21:58:41.190 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.190 INFO:teuthology.orchestra.run.smithi033.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-12-07T21:58:41.191 INFO:teuthology.orchestra.run.smithi033.stdout:Searching for gevent==23.9.1 2023-12-07T21:58:41.191 INFO:teuthology.orchestra.run.smithi033.stdout:Best match: gevent 23.9.1 2023-12-07T21:58:41.191 INFO:teuthology.orchestra.run.smithi033.stdout:Adding gevent 23.9.1 to easy-install.pth file 2023-12-07T21:58:41.191 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.191 INFO:teuthology.orchestra.run.smithi033.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-12-07T21:58:41.191 INFO:teuthology.orchestra.run.smithi033.stdout:Searching for munch==4.0.0 2023-12-07T21:58:41.191 INFO:teuthology.orchestra.run.smithi033.stdout:Best match: munch 4.0.0 2023-12-07T21:58:41.191 INFO:teuthology.orchestra.run.smithi033.stdout:Adding munch 4.0.0 to easy-install.pth file 2023-12-07T21:58:41.192 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.192 INFO:teuthology.orchestra.run.smithi033.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-12-07T21:58:41.192 INFO:teuthology.orchestra.run.smithi033.stdout:Searching for PyYAML==6.0.1 2023-12-07T21:58:41.193 INFO:teuthology.orchestra.run.smithi033.stdout:Best match: PyYAML 6.0.1 2023-12-07T21:58:41.193 INFO:teuthology.orchestra.run.smithi033.stdout:Adding PyYAML 6.0.1 to easy-install.pth file 2023-12-07T21:58:41.193 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.193 INFO:teuthology.orchestra.run.smithi033.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-12-07T21:58:41.193 INFO:teuthology.orchestra.run.smithi033.stdout:Searching for boto3==1.24.96 2023-12-07T21:58:41.193 INFO:teuthology.orchestra.run.smithi033.stdout:Best match: boto3 1.24.96 2023-12-07T21:58:41.193 INFO:teuthology.orchestra.run.smithi033.stdout:Adding boto3 1.24.96 to easy-install.pth file 2023-12-07T21:58:41.193 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.193 INFO:teuthology.orchestra.run.smithi033.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-12-07T21:58:41.194 INFO:teuthology.orchestra.run.smithi033.stdout:Searching for boto==2.49.0 2023-12-07T21:58:41.194 INFO:teuthology.orchestra.run.smithi033.stdout:Best match: boto 2.49.0 2023-12-07T21:58:41.194 INFO:teuthology.orchestra.run.smithi033.stdout:Adding boto 2.49.0 to easy-install.pth file 2023-12-07T21:58:41.194 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.194 INFO:teuthology.orchestra.run.smithi033.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-12-07T21:58:41.194 INFO:teuthology.orchestra.run.smithi033.stdout:Searching for six==1.16.0 2023-12-07T21:58:41.194 INFO:teuthology.orchestra.run.smithi033.stdout:Best match: six 1.16.0 2023-12-07T21:58:41.195 INFO:teuthology.orchestra.run.smithi033.stdout:Adding six 1.16.0 to easy-install.pth file 2023-12-07T21:58:41.195 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.195 INFO:teuthology.orchestra.run.smithi033.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-12-07T21:58:41.196 INFO:teuthology.orchestra.run.smithi033.stdout:Searching for zope.interface==6.1 2023-12-07T21:58:41.196 INFO:teuthology.orchestra.run.smithi033.stdout:Best match: zope.interface 6.1 2023-12-07T21:58:41.196 INFO:teuthology.orchestra.run.smithi033.stdout:Adding zope.interface 6.1 to easy-install.pth file 2023-12-07T21:58:41.196 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.196 INFO:teuthology.orchestra.run.smithi033.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-12-07T21:58:41.196 INFO:teuthology.orchestra.run.smithi033.stdout:Searching for zope.event==5.0 2023-12-07T21:58:41.196 INFO:teuthology.orchestra.run.smithi033.stdout:Best match: zope.event 5.0 2023-12-07T21:58:41.196 INFO:teuthology.orchestra.run.smithi033.stdout:Adding zope.event 5.0 to easy-install.pth file 2023-12-07T21:58:41.196 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.196 INFO:teuthology.orchestra.run.smithi033.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-12-07T21:58:41.197 INFO:teuthology.orchestra.run.smithi033.stdout:Searching for greenlet==3.0.1 2023-12-07T21:58:41.197 INFO:teuthology.orchestra.run.smithi033.stdout:Best match: greenlet 3.0.1 2023-12-07T21:58:41.197 INFO:teuthology.orchestra.run.smithi033.stdout:Adding greenlet 3.0.1 to easy-install.pth file 2023-12-07T21:58:41.197 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.198 INFO:teuthology.orchestra.run.smithi033.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-12-07T21:58:41.198 INFO:teuthology.orchestra.run.smithi033.stdout:Searching for s3transfer==0.6.2 2023-12-07T21:58:41.198 INFO:teuthology.orchestra.run.smithi033.stdout:Best match: s3transfer 0.6.2 2023-12-07T21:58:41.198 INFO:teuthology.orchestra.run.smithi033.stdout:Adding s3transfer 0.6.2 to easy-install.pth file 2023-12-07T21:58:41.198 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.198 INFO:teuthology.orchestra.run.smithi033.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-12-07T21:58:41.199 INFO:teuthology.orchestra.run.smithi033.stdout:Searching for jmespath==1.0.1 2023-12-07T21:58:41.199 INFO:teuthology.orchestra.run.smithi033.stdout:Best match: jmespath 1.0.1 2023-12-07T21:58:41.200 INFO:teuthology.orchestra.run.smithi033.stdout:Adding jmespath 1.0.1 to easy-install.pth file 2023-12-07T21:58:41.200 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.200 INFO:teuthology.orchestra.run.smithi033.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-12-07T21:58:41.200 INFO:teuthology.orchestra.run.smithi033.stdout:Searching for botocore==1.27.96 2023-12-07T21:58:41.200 INFO:teuthology.orchestra.run.smithi033.stdout:Best match: botocore 1.27.96 2023-12-07T21:58:41.200 INFO:teuthology.orchestra.run.smithi033.stdout:Adding botocore 1.27.96 to easy-install.pth file 2023-12-07T21:58:41.200 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.200 INFO:teuthology.orchestra.run.smithi033.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-12-07T21:58:41.200 INFO:teuthology.orchestra.run.smithi033.stdout:Searching for setuptools==44.1.0 2023-12-07T21:58:41.200 INFO:teuthology.orchestra.run.smithi033.stdout:Best match: setuptools 44.1.0 2023-12-07T21:58:41.200 INFO:teuthology.orchestra.run.smithi033.stdout:Adding setuptools 44.1.0 to easy-install.pth file 2023-12-07T21:58:41.201 INFO:teuthology.orchestra.run.smithi033.stdout:Installing easy_install script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2023-12-07T21:58:41.201 INFO:teuthology.orchestra.run.smithi033.stdout:Installing easy_install-3.8 script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2023-12-07T21:58:41.201 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.201 INFO:teuthology.orchestra.run.smithi033.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-12-07T21:58:41.201 INFO:teuthology.orchestra.run.smithi033.stdout:Searching for urllib3==1.26.18 2023-12-07T21:58:41.201 INFO:teuthology.orchestra.run.smithi033.stdout:Best match: urllib3 1.26.18 2023-12-07T21:58:41.201 INFO:teuthology.orchestra.run.smithi033.stdout:Adding urllib3 1.26.18 to easy-install.pth file 2023-12-07T21:58:41.202 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.202 INFO:teuthology.orchestra.run.smithi033.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-12-07T21:58:41.202 INFO:teuthology.orchestra.run.smithi033.stdout:Searching for python-dateutil==2.8.2 2023-12-07T21:58:41.202 INFO:teuthology.orchestra.run.smithi033.stdout:Best match: python-dateutil 2.8.2 2023-12-07T21:58:41.203 INFO:teuthology.orchestra.run.smithi033.stdout:Adding python-dateutil 2.8.2 to easy-install.pth file 2023-12-07T21:58:41.203 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T21:58:41.203 INFO:teuthology.orchestra.run.smithi033.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-12-07T21:58:41.203 INFO:teuthology.orchestra.run.smithi033.stdout:Finished processing dependencies for s3tests==0.0.1 2023-12-07T21:58:41.221 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:58:41.222 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2023-12-07T21:58:41.228 INFO:tasks.s3tests:Configuring boto... 2023-12-07T21:58:41.229 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-07T21:58:41.229 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/boto.cfg 2023-12-07T21:58:41.282 DEBUG:teuthology.orchestra.run.smithi033:s3 tests against rgw> S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain' 2023-12-07T21:58:41.677 INFO:teuthology.orchestra.run.smithi033.stderr:/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_headers.py:126: SyntaxWarning: "is" with a literal. Did you mean "=="? 2023-12-07T21:58:41.677 INFO:teuthology.orchestra.run.smithi033.stderr: if boto_type is 'S3Connection': 2023-12-07T21:58:41.677 INFO:teuthology.orchestra.run.smithi033.stderr:/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_headers.py:131: SyntaxWarning: "is" with a literal. Did you mean "=="? 2023-12-07T21:58:41.677 INFO:teuthology.orchestra.run.smithi033.stderr: elif boto_type is 'HTTPRequest': 2023-12-07T21:58:41.677 INFO:teuthology.orchestra.run.smithi033.stderr:/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3_website.py:191: SyntaxWarning: "is" with a literal. Did you mean "=="? 2023-12-07T21:58:41.677 INFO:teuthology.orchestra.run.smithi033.stderr: if k is 'BucketName': 2023-12-07T21:58:41.695 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok 2023-12-07T21:58:41.712 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok 2023-12-07T21:58:41.732 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_date_and_amz_date ... ok 2023-12-07T21:58:41.793 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok 2023-12-07T21:58:41.818 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_none ... ok 2023-12-07T21:58:41.838 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_contentlength_none ... ok 2023-12-07T21:58:41.873 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_acl_create_contentlength_none ... ok 2023-12-07T21:58:41.893 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok 2023-12-07T21:58:41.937 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_empty ... ok 2023-12-07T21:58:41.940 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_none ... ok 2023-12-07T21:58:41.961 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws2 ... ok 2023-12-07T21:58:41.980 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws2 ... ok 2023-12-07T21:58:41.998 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws2 ... ok 2023-12-07T21:58:42.015 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws2 ... ok 2023-12-07T21:58:42.017 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws2 ... ok 2023-12-07T21:58:42.019 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws2 ... ok 2023-12-07T21:58:42.019 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws4 ... SKIP 2023-12-07T21:58:42.019 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws4 ... SKIP 2023-12-07T21:58:42.020 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws4 ... SKIP 2023-12-07T21:58:42.020 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws4 ... SKIP 2023-12-07T21:58:42.020 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_empty_aws4 ... SKIP 2023-12-07T21:58:42.020 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_none_aws4 ... SKIP 2023-12-07T21:58:42.021 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_date_invalid_aws4 ... SKIP 2023-12-07T21:58:42.021 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_invalid_aws4 ... SKIP 2023-12-07T21:58:42.021 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_date_empty_aws4 ... SKIP 2023-12-07T21:58:42.021 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_empty_aws4 ... SKIP 2023-12-07T21:58:42.021 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws4 ... SKIP 2023-12-07T21:58:42.022 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_none_aws4 ... SKIP 2023-12-07T21:58:42.022 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_today_aws4 ... SKIP 2023-12-07T21:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_today_aws4 ... SKIP 2023-12-07T21:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_today_aws4 ... SKIP 2023-12-07T21:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_today_aws4 ... SKIP 2023-12-07T21:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_epoch_aws4 ... SKIP 2023-12-07T21:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-12-07T21:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_end_aws4 ... SKIP 2023-12-07T21:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_end_aws4 ... SKIP 2023-12-07T21:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_custom_header_aws4 ... SKIP 2023-12-07T21:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_header_aws4 ... SKIP 2023-12-07T21:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws4 ... SKIP 2023-12-07T21:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_empty_aws4 ... SKIP 2023-12-07T21:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_none_aws4 ... SKIP 2023-12-07T21:58:42.023 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_invalid_aws4 ... SKIP 2023-12-07T21:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_invalid_aws4 ... SKIP 2023-12-07T21:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_empty_aws4 ... SKIP 2023-12-07T21:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_empty_aws4 ... SKIP 2023-12-07T21:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws4 ... SKIP 2023-12-07T21:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_none_aws4 ... SKIP 2023-12-07T21:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_today_aws4 ... SKIP 2023-12-07T21:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_today_aws4 ... SKIP 2023-12-07T21:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_after_today_aws4 ... SKIP 2023-12-07T21:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_after_today_aws4 ... SKIP 2023-12-07T21:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws4 ... SKIP 2023-12-07T21:58:42.025 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-12-07T21:58:42.085 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_versioning_obj_read_not_exist_null ... ok 2023-12-07T21:58:42.085 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_lifecycle_transition ... SKIP 2023-12-07T21:58:42.085 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_lifecycle_transition_single_rule_multi_trans ... SKIP 2023-12-07T21:58:42.085 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_lifecycle_set_noncurrent_transition ... SKIP 2023-12-07T21:58:42.085 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_object_storage_class ... SKIP 2023-12-07T21:58:42.086 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_object_storage_class_multipart ... SKIP 2023-12-07T21:58:42.086 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_object_modify_storage_class ... SKIP 2023-12-07T21:58:42.086 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_object_modify_storage_class_multipart ... SKIP 2023-12-07T21:58:42.086 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_object_storage_class_copy ... SKIP 2023-12-07T21:58:42.087 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_object_storage_class_copy_multipart ... SKIP 2023-12-07T21:58:42.190 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_atomic_dual_conditional_write_1mb ... ok 2023-12-07T21:58:42.274 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_atomic_write_bucket_gone ... ok 2023-12-07T21:58:44.056 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_1 ... ok 2023-12-07T21:58:44.125 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_2 ... ok 2023-12-07T21:58:44.170 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_bucket_policy_set_condition_operator_end_with_IfExists ... ok 2023-12-07T21:58:44.251 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_enc ... ok 2023-12-07T21:58:44.285 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_request_obj_tag ... ok 2023-12-07T21:58:44.422 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.check_can_test_website ... SKIP 2023-12-07T21:58:44.422 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_nonexistant_bucket_rgw ... SKIP 2023-12-07T21:58:44.422 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_public_index ... SKIP 2023-12-07T21:58:44.422 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_public_index ... SKIP 2023-12-07T21:58:44.422 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty ... SKIP 2023-12-07T21:58:44.426 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty ... SKIP 2023-12-07T21:58:44.427 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index ... SKIP 2023-12-07T21:58:44.427 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index ... SKIP 2023-12-07T21:58:44.427 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_missingerrordoc ... SKIP 2023-12-07T21:58:44.427 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_missingerrordoc ... SKIP 2023-12-07T21:58:44.427 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_missingerrordoc ... SKIP 2023-12-07T21:58:44.427 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_missingerrordoc ... SKIP 2023-12-07T21:58:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_blockederrordoc ... SKIP 2023-12-07T21:58:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_pubilc_errordoc ... SKIP 2023-12-07T21:58:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_blockederrordoc ... SKIP 2023-12-07T21:58:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_blockederrordoc ... SKIP 2023-12-07T21:58:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_blockederrordoc ... SKIP 2023-12-07T21:58:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_gooderrordoc ... SKIP 2023-12-07T21:58:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_gooderrordoc ... SKIP 2023-12-07T21:58:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_gooderrordoc ... SKIP 2023-12-07T21:58:44.428 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_gooderrordoc ... SKIP 2023-12-07T21:58:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_base ... SKIP 2023-12-07T21:58:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path ... SKIP 2023-12-07T21:58:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path_upgrade ... SKIP 2023-12-07T21:58:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_xredirect_nonwebsite ... SKIP 2023-12-07T21:58:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_relative ... SKIP 2023-12-07T21:58:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_abs ... SKIP 2023-12-07T21:58:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_relative ... SKIP 2023-12-07T21:58:44.429 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_abs ... SKIP 2023-12-07T21:58:44.430 INFO:teuthology.orchestra.run.smithi033.stderr:SKIP 2023-12-07T21:58:47.443 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests.functional.test_utils.test_generate ... ok 2023-12-07T21:58:47.481 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok 2023-12-07T21:58:57.817 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_bad ... ok 2023-12-07T21:58:57.855 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_empty ... ok 2023-12-07T21:58:57.915 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_none ... ok 2023-12-07T21:58:57.982 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_mismatch ... ok 2023-12-07T21:58:58.045 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_empty ... ok 2023-12-07T21:58:58.107 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_none ... ok 2023-12-07T21:58:58.145 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contentlength_negative ... ok 2023-12-07T21:58:58.201 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok 2023-12-07T21:58:58.246 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_empty ... ok 2023-12-07T21:58:58.319 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_none ... ok 2023-12-07T21:58:58.352 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_bucket_put_bad_canned_acl ... ok 2023-12-07T21:58:58.377 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_mismatch ... ok 2023-12-07T21:58:58.404 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_empty ... ok 2023-12-07T21:58:58.414 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_contentlength_negative ... ok 2023-12-07T21:58:58.449 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws2 ... ok 2023-12-07T21:58:58.505 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_empty_aws2 ... ok 2023-12-07T21:58:58.562 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_none_aws2 ... ok 2023-12-07T21:58:58.602 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_invalid_aws2 ... ok 2023-12-07T21:58:58.644 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_empty_aws2 ... ok 2023-12-07T21:58:58.680 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_today_aws2 ... ok 2023-12-07T21:58:58.716 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_epoch_aws2 ... ok 2023-12-07T21:58:58.749 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_after_end_aws2 ... ok 2023-12-07T21:58:58.776 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_empty_aws2 ... ok 2023-12-07T21:58:58.802 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_none_aws2 ... ok 2023-12-07T21:58:58.813 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_invalid_aws2 ... ok 2023-12-07T21:58:58.821 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_empty_aws2 ... ok 2023-12-07T21:58:58.829 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_today_aws2 ... ok 2023-12-07T21:58:58.837 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_after_today_aws2 ... ok 2023-12-07T21:58:58.845 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws2 ... ok 2023-12-07T21:58:58.897 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_empty ... ok 2023-12-07T21:58:58.971 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_distinct ... ok 2023-12-07T21:58:59.051 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_many ... ok 2023-12-07T21:58:59.136 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_many ... ok 2023-12-07T21:58:59.264 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_basic_key_count ... ok 2023-12-07T21:58:59.363 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_basic ... ok 2023-12-07T21:58:59.449 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_basic ... ok 2023-12-07T21:58:59.536 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_encoding_basic ... ok 2023-12-07T21:58:59.658 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_encoding_basic ... ok 2023-12-07T21:58:59.842 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix ... ok 2023-12-07T21:59:00.023 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix ... ok 2023-12-07T21:59:00.073 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_ends_with_delimiter ... ok 2023-12-07T21:59:00.117 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_ends_with_delimiter ... ok 2023-12-07T21:59:00.204 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_alt ... ok 2023-12-07T21:59:00.292 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_alt ... ok 2023-12-07T21:59:00.462 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_underscore ... ok 2023-12-07T21:59:00.692 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_underscore ... ok 2023-12-07T21:59:00.782 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_percentage ... ok 2023-12-07T21:59:00.863 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_percentage ... ok 2023-12-07T21:59:00.948 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_whitespace ... ok 2023-12-07T21:59:01.039 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_whitespace ... ok 2023-12-07T21:59:01.132 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_dot ... ok 2023-12-07T21:59:01.220 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_dot ... ok 2023-12-07T21:59:01.313 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_unreadable ... ok 2023-12-07T21:59:01.410 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_unreadable ... ok 2023-12-07T21:59:01.502 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_empty ... ok 2023-12-07T21:59:01.594 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_empty ... ok 2023-12-07T21:59:01.696 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_none ... ok 2023-12-07T21:59:01.790 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_none ... ok 2023-12-07T21:59:01.876 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_notempty ... ok 2023-12-07T21:59:01.954 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_defaultempty ... ok 2023-12-07T21:59:02.067 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_empty ... ok 2023-12-07T21:59:02.158 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_exist ... ok 2023-12-07T21:59:02.256 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_not_exist ... ok 2023-12-07T21:59:14.803 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_skip_special ... ok 2023-12-07T21:59:14.883 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_basic ... ok 2023-12-07T21:59:14.961 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_basic ... ok 2023-12-07T21:59:15.039 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_alt ... ok 2023-12-07T21:59:15.110 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_alt ... ok 2023-12-07T21:59:15.224 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_empty ... ok 2023-12-07T21:59:15.303 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_empty ... ok 2023-12-07T21:59:15.385 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_none ... ok 2023-12-07T21:59:15.461 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_none ... ok 2023-12-07T21:59:15.550 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_not_exist ... ok 2023-12-07T21:59:15.638 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_not_exist ... ok 2023-12-07T21:59:15.725 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_unreadable ... ok 2023-12-07T21:59:15.809 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_unreadable ... ok 2023-12-07T21:59:15.899 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_basic ... ok 2023-12-07T21:59:15.992 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_basic ... ok 2023-12-07T21:59:16.093 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_alt ... ok 2023-12-07T21:59:16.186 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_alt ... ok 2023-12-07T21:59:16.276 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_not_exist ... ok 2023-12-07T21:59:16.371 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_not_exist ... ok 2023-12-07T21:59:16.462 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_delimiter_not_exist ... ok 2023-12-07T21:59:16.586 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_delimiter_not_exist ... ok 2023-12-07T21:59:16.674 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-12-07T21:59:16.758 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-12-07T21:59:16.861 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_one ... ok 2023-12-07T21:59:16.957 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_one ... ok 2023-12-07T21:59:17.047 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_zero ... ok 2023-12-07T21:59:17.128 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_zero ... ok 2023-12-07T21:59:17.230 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_none ... ok 2023-12-07T21:59:17.333 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_none ... ok 2023-12-07T21:59:17.742 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_account_usage ... ok 2023-12-07T21:59:17.788 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_head_bucket_usage ... ok 2023-12-07T21:59:18.243 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_unordered ... ok 2023-12-07T21:59:18.726 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_unordered ... ok 2023-12-07T21:59:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_invalid ... ok 2023-12-07T21:59:18.916 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_none ... ok 2023-12-07T21:59:19.005 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_empty ... ok 2023-12-07T21:59:19.093 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken_empty ... ok 2023-12-07T21:59:19.188 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken ... ok 2023-12-07T21:59:19.293 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_both_continuationtoken_startafter ... ok 2023-12-07T21:59:19.387 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_unreadable ... ok 2023-12-07T21:59:19.481 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_unreadable ... ok 2023-12-07T21:59:19.572 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_not_in_list ... ok 2023-12-07T21:59:19.666 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_not_in_list ... ok 2023-12-07T21:59:19.754 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_after_list ... ok 2023-12-07T21:59:19.843 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_after_list ... ok 2023-12-07T21:59:19.943 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data ... ok 2023-12-07T21:59:20.189 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data_versioning ... ok 2023-12-07T21:59:20.241 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous ... ok 2023-12-07T21:59:20.290 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous ... ok 2023-12-07T21:59:20.327 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous_fail ... ok 2023-12-07T21:59:20.365 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous_fail ... ok 2023-12-07T21:59:20.375 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_notexist ... ok 2023-12-07T21:59:20.384 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_notexist ... ok 2023-12-07T21:59:20.394 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_notexist ... ok 2023-12-07T21:59:20.464 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_nonempty ... ok 2023-12-07T21:59:20.686 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_concurrent_set_canned_acl ... ok 2023-12-07T21:59:20.696 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_write_to_nonexist_bucket ... ok 2023-12-07T21:59:20.802 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_delete ... ok 2023-12-07T21:59:20.846 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_read_not_exist ... ok 2023-12-07T21:59:20.883 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_requestid_matches_header_on_error ... ok 2023-12-07T21:59:21.043 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete ... ok 2023-12-07T21:59:21.223 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete ... ok 2023-12-07T21:59:21.273 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_head_zero_bytes ... ok 2023-12-07T21:59:21.319 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_write_check_etag ... ok 2023-12-07T21:59:21.373 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_write_cache_control ... ok 2023-12-07T21:59:21.425 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_write_expires ... ok 2023-12-07T21:59:21.503 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_write_read_update_read_delete ... ok 2023-12-07T21:59:21.559 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_good ... ok 2023-12-07T21:59:21.609 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_empty ... ok 2023-12-07T21:59:21.694 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_overwrite_to_empty ... ok 2023-12-07T21:59:21.761 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_metadata_replaced_on_put ... ok 2023-12-07T21:59:21.812 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_write_file ... ok 2023-12-07T21:59:21.856 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_anonymous_request ... ok 2023-12-07T21:59:21.940 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request ... ok 2023-12-07T21:59:21.984 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_no_content_type ... ok 2023-12-07T21:59:22.015 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request_bad_access_key ... ok 2023-12-07T21:59:22.051 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_success_code ... ok 2023-12-07T21:59:22.089 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_invalid_success_code ... ok 2023-12-07T21:59:22.197 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_larger_than_chunk ... ok 2023-12-07T21:59:22.247 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_key_from_filename ... ok 2023-12-07T21:59:22.289 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_ignored_header ... ok 2023-12-07T21:59:22.329 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_case_insensitive_condition_fields ... ok 2023-12-07T21:59:22.383 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_escaped_field_values ... ok 2023-12-07T21:59:22.434 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_success_redirect_action ... ok 2023-12-07T21:59:22.470 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_signature ... ok 2023-12-07T21:59:22.506 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_access_key ... ok 2023-12-07T21:59:22.544 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_date_format ... ok 2023-12-07T21:59:22.575 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_no_key_specified ... ok 2023-12-07T21:59:22.608 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_signature ... ok 2023-12-07T21:59:22.644 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_policy_condition ... ok 2023-12-07T21:59:22.688 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_user_specified_header ... ok 2023-12-07T21:59:22.723 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_request_missing_policy_specified_field ... ok 2023-12-07T21:59:22.763 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_condition_is_case_sensitive ... ok 2023-12-07T21:59:22.800 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_expires_is_case_sensitive ... ok 2023-12-07T21:59:22.842 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_expired_policy ... ok 2023-12-07T21:59:22.882 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_request_field_value ... ok 2023-12-07T21:59:22.915 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_expires_condition ... ok 2023-12-07T21:59:22.946 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_conditions_list ... ok 2023-12-07T21:59:22.981 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_limit_exceeded ... ok 2023-12-07T21:59:23.015 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_content_length_argument ... ok 2023-12-07T21:59:23.048 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_content_length_argument ... ok 2023-12-07T21:59:23.084 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_below_minimum ... ok 2023-12-07T21:59:23.121 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_empty_conditions ... ok 2023-12-07T21:59:23.173 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_good ... ok 2023-12-07T21:59:23.235 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_failed ... ok 2023-12-07T21:59:23.293 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_good ... ok 2023-12-07T21:59:23.349 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_failed ... ok 2023-12-07T21:59:23.400 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_good ... ok 2023-12-07T21:59:24.465 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_failed ... ok 2023-12-07T21:59:24.526 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_good ... ok 2023-12-07T21:59:24.575 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_failed ... ok 2023-12-07T21:59:24.686 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_good ... ok 2023-12-07T21:59:24.811 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_failed ... ok 2023-12-07T21:59:24.934 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_overwrite_existed_good ... ok 2023-12-07T21:59:25.013 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_nonexisted_failed ... ok 2023-12-07T21:59:25.117 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_good ... ok 2023-12-07T21:59:25.190 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_failed ... ok 2023-12-07T21:59:25.254 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_nonexisted_good ... ok 2023-12-07T21:59:25.320 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_overwrite_existed_failed ... ok 2023-12-07T21:59:25.373 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get ... ok 2023-12-07T21:59:25.516 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_gone ... ok 2023-12-07T21:59:25.651 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_delete_key_bucket_gone ... ok 2023-12-07T21:59:25.719 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_gone ... ok 2023-12-07T21:59:25.759 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_head ... ok 2023-12-07T21:59:25.769 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_notexist ... ok 2023-12-07T21:59:25.870 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_extended ... ok 2023-12-07T21:59:25.920 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_acl ... ok 2023-12-07T21:59:25.971 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_acl ... ok 2023-12-07T21:59:26.022 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated ... ok 2023-12-07T21:59:26.082 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_response_headers ... ok 2023-12-07T21:59:26.135 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_acl ... ok 2023-12-07T21:59:26.189 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_acl ... ok 2023-12-07T21:59:26.319 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_gone ... ok 2023-12-07T21:59:26.384 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_gone ... ok 2023-12-07T21:59:26.433 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired ... ok 2023-12-07T21:59:26.484 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired_tenant ... ok 2023-12-07T21:59:26.535 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_range_zero ... ok 2023-12-07T21:59:26.585 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_max_range ... ok 2023-12-07T21:59:26.633 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_positive_range ... ok 2023-12-07T21:59:26.687 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put ... ok 2023-12-07T21:59:26.747 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put_write_access ... ok 2023-12-07T21:59:26.824 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_put_authenticated ... ok 2023-12-07T21:59:26.874 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_raw_put_authenticated_expired ... ok 2023-12-07T21:59:26.889 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_one ... ok 2023-12-07T21:59:26.905 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_two ... ok 2023-12-07T21:59:26.921 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_ip ... ok 2023-12-07T21:59:26.938 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_dns_underscore ... ok 2023-12-07T21:59:26.982 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists ... ok 2023-12-07T21:59:27.021 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_get_location ... ok 2023-12-07T21:59:27.065 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists_nonowner ... ok 2023-12-07T21:59:27.108 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_overwrite_acl ... FAIL 2023-12-07T21:59:27.147 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_new_acl ... FAIL 2023-12-07T21:59:27.184 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_default ... ok 2023-12-07T21:59:27.218 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_during_create ... ok 2023-12-07T21:59:27.263 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned ... ok 2023-12-07T21:59:27.297 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_publicreadwrite ... ok 2023-12-07T21:59:27.332 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_authenticatedread ... ok 2023-12-07T21:59:27.386 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_acl_default ... ok 2023-12-07T21:59:27.440 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_during_create ... ok 2023-12-07T21:59:27.546 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned ... ok 2023-12-07T21:59:27.608 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_publicreadwrite ... ok 2023-12-07T21:59:27.658 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_authenticatedread ... ok 2023-12-07T21:59:27.733 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerread ... ok 2023-12-07T21:59:27.822 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerfullcontrol ... ok 2023-12-07T21:59:27.898 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_owner ... ok 2023-12-07T21:59:27.981 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_attributes ... ok 2023-12-07T21:59:28.025 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_private_to_private ... ok 2023-12-07T21:59:28.138 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_acl ... ok 2023-12-07T21:59:28.215 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_acl_write ... ok 2023-12-07T21:59:28.290 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_acl_writeacp ... ok 2023-12-07T21:59:28.406 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_acl_read ... ok 2023-12-07T21:59:28.481 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_acl_readacp ... ok 2023-12-07T21:59:28.595 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_fullcontrol ... ok 2023-12-07T21:59:28.689 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_read ... ok 2023-12-07T21:59:28.783 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_readacp ... ok 2023-12-07T21:59:28.909 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_write ... ok 2023-12-07T21:59:29.007 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_writeacp ... ok 2023-12-07T21:59:29.053 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_nonexist_user ... ok 2023-12-07T21:59:29.145 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_no_grants ... ok 2023-12-07T21:59:29.198 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_header_acl_grants ... ok 2023-12-07T21:59:29.260 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_header_acl_grants ... ok 2023-12-07T21:59:29.318 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email ... ok 2023-12-07T21:59:29.371 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email_not_exist ... ok 2023-12-07T21:59:29.499 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_revoke_all ... ok 2023-12-07T21:59:29.625 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_private ... ok 2023-12-07T21:59:29.794 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_private ... ok 2023-12-07T21:59:29.931 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicread ... ok 2023-12-07T21:59:30.058 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicread ... ok 2023-12-07T21:59:30.219 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicreadwrite ... ok 2023-12-07T21:59:30.336 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicreadwrite ... ok 2023-12-07T21:59:30.461 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_private ... ok 2023-12-07T21:59:30.593 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicread ... ok 2023-12-07T21:59:30.726 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicreadwrite ... ok 2023-12-07T21:59:30.917 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_private ... ok 2023-12-07T21:59:31.122 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicread ... ok 2023-12-07T21:59:31.313 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicreadwrite ... ok 2023-12-07T21:59:31.589 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_buckets_create_then_list ... ok 2023-12-07T21:59:31.595 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_anonymous ... ok 2023-12-07T21:59:31.603 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_invalid_auth ... ok 2023-12-07T21:59:31.612 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_bad_auth ... ok 2023-12-07T21:59:31.810 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_alpha ... ok 2023-12-07T21:59:31.997 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_digit ... ok 2023-12-07T21:59:32.034 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_period ... ok 2023-12-07T21:59:32.099 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_hyphen ... ok 2023-12-07T21:59:32.192 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_not_overriding ... ok 2023-12-07T21:59:32.808 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_special_key_names ... ok 2023-12-07T21:59:32.930 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_special_prefix ... ok 2023-12-07T21:59:33.031 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_copy_zero_size ... ok 2023-12-07T21:59:33.114 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_copy_same_bucket ... ok 2023-12-07T21:59:33.247 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_copy_verify_contenttype ... ok 2023-12-07T21:59:33.364 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself ... ok 2023-12-07T21:59:33.439 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself_with_metadata ... ok 2023-12-07T21:59:33.598 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_copy_diff_bucket ... ok 2023-12-07T21:59:33.682 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_bucket ... ok 2023-12-07T21:59:33.821 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_object_bucket ... ok 2023-12-07T21:59:33.917 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_copy_canned_acl ... ok 2023-12-07T21:59:34.089 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_copy_retaining_metadata ... ok 2023-12-07T21:59:34.261 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_copy_replacing_metadata ... ok 2023-12-07T21:59:34.344 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_copy_bucket_not_found ... ok 2023-12-07T21:59:34.395 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_copy_key_not_found ... ok 2023-12-07T21:59:34.810 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_bucket ... ok 2023-12-07T21:59:34.939 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_url_encoding ... ok 2023-12-07T21:59:36.266 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioning_multipart_upload ... ok 2023-12-07T21:59:36.325 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_empty ... ok 2023-12-07T21:59:36.420 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_small ... ok 2023-12-07T21:59:36.721 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_small ... ok 2023-12-07T21:59:36.800 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_invalid_range ... ok 2023-12-07T21:59:36.979 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_without_range ... ok 2023-12-07T21:59:37.975 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_special_names ... ok 2023-12-07T21:59:38.992 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload ... ok 2023-12-07T21:59:40.783 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_versioned ... ok 2023-12-07T21:59:46.916 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_resend_part ... ok 2023-12-07T21:59:48.207 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_multiple_sizes ... ok 2023-12-07T21:59:49.804 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_multiple_sizes ... ok 2023-12-07T21:59:50.048 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_size_too_small ... ok 2023-12-07T21:59:50.576 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_contents ... ok 2023-12-07T21:59:51.017 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_overwrite_existing_object ... ok 2023-12-07T21:59:51.281 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload ... ok 2023-12-07T21:59:51.343 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload_not_found ... ok 2023-12-07T21:59:51.796 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload ... ok 2023-12-07T21:59:51.882 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_missing_part ... ok 2023-12-07T21:59:51.970 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_incorrect_etag ... ok 2023-12-07T21:59:52.018 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_100_continue ... ok 2023-12-07T21:59:52.130 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_set_cors ... ok 2023-12-07T21:59:55.298 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_response ... ok 2023-12-07T21:59:58.363 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_wildcard ... ok 2023-12-07T22:00:01.422 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_cors_header_option ... ok 2023-12-07T22:00:01.492 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_cors_presigned_get_object ... ok 2023-12-07T22:00:01.549 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_cors_presigned_get_object_tenant ... ok 2023-12-07T22:00:01.658 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_set_bucket_tagging ... ok 2023-12-07T22:00:01.861 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_1mb ... ok 2023-12-07T22:00:02.131 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_4mb ... ok 2023-12-07T22:00:02.537 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_8mb ... ok 2023-12-07T22:00:02.790 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_1mb ... ok 2023-12-07T22:00:03.195 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_4mb ... ok 2023-12-07T22:00:03.815 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_8mb ... ok 2023-12-07T22:00:04.022 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_1mb ... ok 2023-12-07T22:00:04.406 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_4mb ... ok 2023-12-07T22:00:05.043 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_8mb ... ok 2023-12-07T22:00:05.301 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_atomic_conditional_write_1mb ... ok 2023-12-07T22:00:05.438 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_atomic_multipart_upload_write ... ok 2023-12-07T22:00:05.681 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_multipart_resend_first_finishes_last ... ok 2023-12-07T22:00:05.751 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_response_code ... ok 2023-12-07T22:00:09.588 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_ranged_big_request_response_code ... ok 2023-12-07T22:00:09.647 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_skip_leading_bytes_response_code ... ok 2023-12-07T22:00:09.702 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_return_trailing_bytes_response_code ... ok 2023-12-07T22:00:09.758 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_invalid_range ... ok 2023-12-07T22:00:09.812 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_empty_object ... ok 2023-12-07T22:00:09.911 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_create_suspend ... ok 2023-12-07T22:00:14.729 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove ... ok 2023-12-07T22:00:15.325 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove_head ... ok 2023-12-07T22:00:15.449 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_removal ... ok 2023-12-07T22:00:15.666 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite ... ok 2023-12-07T22:00:15.922 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite_suspended ... ok 2023-12-07T22:00:17.652 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_suspend_versions ... ok 2023-12-07T22:00:19.621 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_all ... ok 2023-12-07T22:00:27.126 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_special_names ... ok 2023-12-07T22:00:29.036 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_overwrite_multipart ... ok 2023-12-07T22:00:29.491 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_list_marker ... ok 2023-12-07T22:00:30.039 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_copy_obj_version ... ok 2023-12-07T22:00:30.359 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete ... ok 2023-12-07T22:00:30.773 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker ... ok 2023-12-07T22:00:30.858 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker_create ... ok 2023-12-07T22:00:31.215 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl ... ok 2023-12-07T22:00:31.520 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl_no_version_specified ... ok 2023-12-07T22:00:31.905 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioned_concurrent_object_create_and_remove ... ok 2023-12-07T22:00:31.962 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set ... ok 2023-12-07T22:00:32.033 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get ... ok 2023-12-07T22:00:32.101 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get_no_id ... ok 2023-12-07T22:00:32.148 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_id_too_long ... ok 2023-12-07T22:00:32.192 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_same_id ... ok 2023-12-07T22:00:32.242 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_invalid_status ... ok 2023-12-07T22:00:32.293 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_date ... ok 2023-12-07T22:00:32.341 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_invalid_date ... ok 2023-12-07T22:00:32.429 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent ... ok 2023-12-07T22:00:32.518 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_deletemarker ... ok 2023-12-07T22:00:32.573 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_filter ... ok 2023-12-07T22:00:32.632 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_empty_filter ... ok 2023-12-07T22:00:32.688 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_multipart ... ok 2023-12-07T22:00:32.750 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1b ... ok 2023-12-07T22:00:32.811 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1kb ... ok 2023-12-07T22:00:32.898 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1MB ... ok 2023-12-07T22:00:32.958 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_13b ... ok 2023-12-07T22:00:33.032 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_method_head ... ok 2023-12-07T22:00:33.093 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_present ... ok 2023-12-07T22:00:33.157 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_other_key ... ok 2023-12-07T22:00:33.203 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_invalid_md5 ... ok 2023-12-07T22:00:33.259 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_md5 ... ok 2023-12-07T22:00:33.311 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_key ... ok 2023-12-07T22:00:33.358 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_encryption_key_no_sse_c ... ok 2023-12-07T22:00:34.570 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_upload ... ok 2023-12-07T22:00:35.344 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_bad_download ... ok 2023-12-07T22:00:35.399 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_post_object_authenticated_request ... ok 2023-12-07T22:00:35.525 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_method_head ... ok 2023-12-07T22:00:35.622 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_present ... ok 2023-12-07T22:00:35.672 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_no_key ... ok 2023-12-07T22:00:35.718 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_not_declared ... ok 2023-12-07T22:00:37.646 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_upload ... ok 2023-12-07T22:00:38.475 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_1 ... ok 2023-12-07T22:00:39.300 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_2 ... ok 2023-12-07T22:00:39.392 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_post_object_authenticated_request ... ok 2023-12-07T22:00:39.481 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1b ... ok 2023-12-07T22:00:39.567 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1kb ... ok 2023-12-07T22:00:39.680 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1MB ... ok 2023-12-07T22:00:39.775 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_13b ... ok 2023-12-07T22:00:39.838 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_read_declare ... ok 2023-12-07T22:00:39.911 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy ... ok 2023-12-07T22:00:39.989 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy ... ok 2023-12-07T22:00:40.079 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_acl ... ok 2023-12-07T22:00:40.180 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_acl ... ok 2023-12-07T22:00:40.373 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_another_bucket ... ok 2023-12-07T22:00:40.601 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_another_bucket ... ok 2023-12-07T22:00:40.794 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_obj_tagging ... ok 2023-12-07T22:00:40.973 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_obj_head_tagging ... ok 2023-12-07T22:00:41.146 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_max_tags ... ok 2023-12-07T22:00:41.310 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_excess_tags ... ok 2023-12-07T22:00:41.483 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_max_kvsize_tags ... ok 2023-12-07T22:00:41.653 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_excess_key_tags ... ok 2023-12-07T22:00:41.822 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_excess_val_tags ... ok 2023-12-07T22:00:42.063 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_modify_tags ... ok 2023-12-07T22:00:42.299 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_delete_tags ... ok 2023-12-07T22:00:42.366 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_anonymous_request ... ok 2023-12-07T22:00:42.422 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_authenticated_request ... ok 2023-12-07T22:00:42.492 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_obj_with_tags ... ok 2023-12-07T22:00:42.676 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_tags_acl_public ... ok 2023-12-07T22:00:42.861 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_tags_acl_public ... ok 2023-12-07T22:00:43.062 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_delete_tags_obj_public ... ok 2023-12-07T22:00:43.247 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_atomic_upload_return_version_id ... ok 2023-12-07T22:00:45.371 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_multipart_upload_return_version_id ... ok 2023-12-07T22:00:45.522 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_existing_tag ... ok 2023-12-07T22:00:45.685 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_tagging_existing_tag ... ok 2023-12-07T22:00:45.881 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_tagging_existing_tag ... ok 2023-12-07T22:00:46.143 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source ... ok 2023-12-07T22:00:46.316 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source_meta ... ok 2023-12-07T22:00:46.399 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_acl ... ok 2023-12-07T22:00:46.535 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_grant ... ok 2023-12-07T22:00:46.695 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_acl_existing_tag ... ok 2023-12-07T22:00:46.791 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock ... ok 2023-12-07T22:00:46.837 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_bucket ... ok 2023-12-07T22:00:46.888 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_with_days_and_years ... ok 2023-12-07T22:00:46.930 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_days ... ok 2023-12-07T22:00:46.979 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_years ... ok 2023-12-07T22:00:47.018 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_status ... ok 2023-12-07T22:00:47.055 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_suspend_versioning ... ok 2023-12-07T22:00:47.105 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock ... ok 2023-12-07T22:00:47.152 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock_invalid_bucket ... ok 2023-12-07T22:00:47.251 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention ... ok 2023-12-07T22:00:47.308 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_bucket ... ok 2023-12-07T22:00:47.382 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_mode ... ok 2023-12-07T22:00:47.534 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention ... ok 2023-12-07T22:00:47.593 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_invalid_bucket ... ok 2023-12-07T22:00:47.779 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_versionid ... ok 2023-12-07T22:00:47.943 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_override_default_retention ... ok 2023-12-07T22:00:48.119 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_increase_period ... ok 2023-12-07T22:00:48.278 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period ... ok 2023-12-07T22:00:48.459 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period_bypass ... ok 2023-12-07T22:00:48.615 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention ... ok 2023-12-07T22:00:48.696 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold ... ok 2023-12-07T22:00:48.799 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_bucket ... ok 2023-12-07T22:00:48.878 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_status ... ok 2023-12-07T22:00:49.023 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold ... ok 2023-12-07T22:00:49.119 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold_invalid_bucket ... ok 2023-12-07T22:00:49.263 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_on ... ok 2023-12-07T22:00:49.373 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_off ... ok 2023-12-07T22:00:49.514 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_metadata ... ok 2023-12-07T22:00:49.627 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_object_lock_uploading_obj ... ok 2023-12-07T22:00:49.696 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifmatch_good ... ok 2023-12-07T22:00:49.776 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifnonematch_failed ... ok 2023-12-07T22:00:49.805 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_user_policy ... ok 2023-12-07T22:00:49.888 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_policy_status ... ok 2023-12-07T22:00:49.940 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_public_acl_bucket_policy_status ... ok 2023-12-07T22:00:49.995 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_authpublic_acl_bucket_policy_status ... ok 2023-12-07T22:00:50.050 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_publicpolicy_acl_bucket_policy_status ... ok 2023-12-07T22:00:50.108 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_acl_bucket_policy_status ... ok 2023-12-07T22:00:50.158 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_deny_bucket_policy_status ... ok 2023-12-07T22:00:50.206 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_get_default_public_block ... ok 2023-12-07T22:00:50.263 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_put_public_block ... ok 2023-12-07T22:00:50.330 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_block_public_put_bucket_acls ... ok 2023-12-07T22:00:50.387 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_block_public_object_canned_acls ... ok 2023-12-07T22:00:50.441 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_block_public_policy ... ok 2023-12-07T22:00:50.543 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_s3.test_ignore_public_acls ... ok 2023-12-07T22:02:24.911 INFO:teuthology.orchestra.run.smithi033.stderr:s3tests_boto3.functional.test_utils.test_generate ... ok 2023-12-07T22:02:24.912 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:24.912 INFO:teuthology.orchestra.run.smithi033.stderr:====================================================================== 2023-12-07T22:02:24.912 INFO:teuthology.orchestra.run.smithi033.stderr:FAIL: s3tests_boto3.functional.test_s3.test_bucket_recreate_overwrite_acl 2023-12-07T22:02:24.912 INFO:teuthology.orchestra.run.smithi033.stderr:---------------------------------------------------------------------- 2023-12-07T22:02:24.912 INFO:teuthology.orchestra.run.smithi033.stderr:Traceback (most recent call last): 2023-12-07T22:02:24.912 INFO:teuthology.orchestra.run.smithi033.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-12-07T22:02:24.912 INFO:teuthology.orchestra.run.smithi033.stderr: self.test(*self.arg) 2023-12-07T22:02:24.912 INFO:teuthology.orchestra.run.smithi033.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 4439, in test_bucket_recreate_overwrite_acl 2023-12-07T22:02:24.912 INFO:teuthology.orchestra.run.smithi033.stderr: e = assert_raises(ClientError, client.create_bucket, Bucket=bucket_name) 2023-12-07T22:02:24.912 INFO:teuthology.orchestra.run.smithi033.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/utils.py", line 21, in assert_raises 2023-12-07T22:02:24.913 INFO:teuthology.orchestra.run.smithi033.stderr: raise AssertionError("%s not raised" % excName) 2023-12-07T22:02:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:AssertionError: ClientError not raised 2023-12-07T22:02:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:-------------------- >> begin captured logging << -------------------- 2023-12-07T22:02:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2023-12-07T22:02:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2023-12-07T22:02:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7f6b64e56430> 2023-12-07T22:02:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2023-12-07T22:02:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60) 2023-12-07T22:02:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.client: DEBUG: Registering retry handlers for service: s3 2023-12-07T22:02:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2023-12-07T22:02:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2023-12-07T22:02:24.913 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2023-12-07T22:02:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2023-12-07T22:02:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2023-12-07T22:02:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-xxboxwb5lcqae2a-211', 'query_string': {}, 'method': 'PUT', 'headers': {'x-amz-acl': 'public-read', 'User-Agent': 'Boto3/1.24.96 Python/3.8.10 Linux/5.4.0-167-generic Botocore/1.27.96'}, 'body': b'', 'url': 'http://smithi033.front.sepia.ceph.com:80/test-client.0-xxboxwb5lcqae2a-211', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-xxboxwb5lcqae2a-211'}}} 2023-12-07T22:02:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2023-12-07T22:02:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.utils: DEBUG: Using S3 path style addressing. 2023-12-07T22:02:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2023-12-07T22:02:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.auth: DEBUG: CanonicalRequest: 2023-12-07T22:02:24.914 INFO:teuthology.orchestra.run.smithi033.stderr:PUT 2023-12-07T22:02:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:/test-client.0-xxboxwb5lcqae2a-211 2023-12-07T22:02:24.915 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:host:smithi033.front.sepia.ceph.com 2023-12-07T22:02:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:x-amz-acl:public-read 2023-12-07T22:02:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-12-07T22:02:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:x-amz-date:20231207T215927Z 2023-12-07T22:02:24.915 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:host;x-amz-acl;x-amz-content-sha256;x-amz-date 2023-12-07T22:02:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-12-07T22:02:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.auth: DEBUG: StringToSign: 2023-12-07T22:02:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:AWS4-HMAC-SHA256 2023-12-07T22:02:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:20231207T215927Z 2023-12-07T22:02:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:20231207/us-east-1/s3/aws4_request 2023-12-07T22:02:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:bc221bada9d6521fc58f878d1d7a7eb27d2f57303d55d06bf6ad6ad239cca641 2023-12-07T22:02:24.915 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.auth: DEBUG: Signature: 2023-12-07T22:02:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:2bdd61e12e9fa842dcc5fe2245dfbb3cf91ee4e40d96a20f5867357b02f68c1f 2023-12-07T22:02:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2023-12-07T22:02:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.endpoint: DEBUG: Sending http request: 2023-12-07T22:02:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi033.front.sepia.ceph.com:80 2023-12-07T22:02:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:urllib3.connectionpool: DEBUG: http://smithi033.front.sepia.ceph.com:80 "PUT /test-client.0-xxboxwb5lcqae2a-211 HTTP/1.1" 200 0 2023-12-07T22:02:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000e202ac2e32e50942-006572403f-105f-default', 'Content-Length': '0', 'Date': 'Thu, 07 Dec 2023 21:59:27 GMT'} 2023-12-07T22:02:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.parsers: DEBUG: Response body: 2023-12-07T22:02:24.916 INFO:teuthology.orchestra.run.smithi033.stderr:b'' 2023-12-07T22:02:24.917 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2023-12-07T22:02:24.917 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.retryhandler: DEBUG: No retry needed. 2023-12-07T22:02:24.917 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.917 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2023-12-07T22:02:24.917 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.917 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.917 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2023-12-07T22:02:24.917 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2023-12-07T22:02:24.918 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.918 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2023-12-07T22:02:24.918 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2023-12-07T22:02:24.918 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-xxboxwb5lcqae2a-211', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.24.96 Python/3.8.10 Linux/5.4.0-167-generic Botocore/1.27.96'}, 'body': b'', 'url': 'http://smithi033.front.sepia.ceph.com:80/test-client.0-xxboxwb5lcqae2a-211', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-xxboxwb5lcqae2a-211'}}} 2023-12-07T22:02:24.918 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.918 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.918 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2023-12-07T22:02:24.918 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.918 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2023-12-07T22:02:24.918 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.auth: DEBUG: CanonicalRequest: 2023-12-07T22:02:24.918 INFO:teuthology.orchestra.run.smithi033.stderr:PUT 2023-12-07T22:02:24.918 INFO:teuthology.orchestra.run.smithi033.stderr:/test-client.0-xxboxwb5lcqae2a-211 2023-12-07T22:02:24.918 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:24.918 INFO:teuthology.orchestra.run.smithi033.stderr:host:smithi033.front.sepia.ceph.com 2023-12-07T22:02:24.919 INFO:teuthology.orchestra.run.smithi033.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-12-07T22:02:24.919 INFO:teuthology.orchestra.run.smithi033.stderr:x-amz-date:20231207T215927Z 2023-12-07T22:02:24.919 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:24.919 INFO:teuthology.orchestra.run.smithi033.stderr:host;x-amz-content-sha256;x-amz-date 2023-12-07T22:02:24.919 INFO:teuthology.orchestra.run.smithi033.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-12-07T22:02:24.919 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.auth: DEBUG: StringToSign: 2023-12-07T22:02:24.919 INFO:teuthology.orchestra.run.smithi033.stderr:AWS4-HMAC-SHA256 2023-12-07T22:02:24.919 INFO:teuthology.orchestra.run.smithi033.stderr:20231207T215927Z 2023-12-07T22:02:24.919 INFO:teuthology.orchestra.run.smithi033.stderr:20231207/us-east-1/s3/aws4_request 2023-12-07T22:02:24.919 INFO:teuthology.orchestra.run.smithi033.stderr:37de3574c83abf61e7a9701d2755bab6d2ffb4c8483c8efcd3f59a4b464877b8 2023-12-07T22:02:24.919 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.auth: DEBUG: Signature: 2023-12-07T22:02:24.919 INFO:teuthology.orchestra.run.smithi033.stderr:48583975c7fa9a6476ca3ed6df3bb61a941c46f0ad0370372c338ab2b36b56c4 2023-12-07T22:02:24.919 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2023-12-07T22:02:24.919 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.endpoint: DEBUG: Sending http request: 2023-12-07T22:02:24.920 INFO:teuthology.orchestra.run.smithi033.stderr:urllib3.connectionpool: DEBUG: http://smithi033.front.sepia.ceph.com:80 "PUT /test-client.0-xxboxwb5lcqae2a-211 HTTP/1.1" 200 0 2023-12-07T22:02:24.920 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000f38d63ccc0f48c37-006572403f-105f-default', 'Content-Length': '0', 'Date': 'Thu, 07 Dec 2023 21:59:27 GMT'} 2023-12-07T22:02:24.920 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.parsers: DEBUG: Response body: 2023-12-07T22:02:24.920 INFO:teuthology.orchestra.run.smithi033.stderr:b'' 2023-12-07T22:02:24.920 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2023-12-07T22:02:24.920 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.retryhandler: DEBUG: No retry needed. 2023-12-07T22:02:24.920 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.920 INFO:teuthology.orchestra.run.smithi033.stderr:--------------------- >> end captured logging << --------------------- 2023-12-07T22:02:24.920 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:24.920 INFO:teuthology.orchestra.run.smithi033.stderr:====================================================================== 2023-12-07T22:02:24.920 INFO:teuthology.orchestra.run.smithi033.stderr:FAIL: s3tests_boto3.functional.test_s3.test_bucket_recreate_new_acl 2023-12-07T22:02:24.920 INFO:teuthology.orchestra.run.smithi033.stderr:---------------------------------------------------------------------- 2023-12-07T22:02:24.920 INFO:teuthology.orchestra.run.smithi033.stderr:Traceback (most recent call last): 2023-12-07T22:02:24.920 INFO:teuthology.orchestra.run.smithi033.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-12-07T22:02:24.920 INFO:teuthology.orchestra.run.smithi033.stderr: self.test(*self.arg) 2023-12-07T22:02:24.921 INFO:teuthology.orchestra.run.smithi033.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 4453, in test_bucket_recreate_new_acl 2023-12-07T22:02:24.921 INFO:teuthology.orchestra.run.smithi033.stderr: e = assert_raises(ClientError, client.create_bucket, Bucket=bucket_name, ACL='public-read') 2023-12-07T22:02:24.921 INFO:teuthology.orchestra.run.smithi033.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/utils.py", line 21, in assert_raises 2023-12-07T22:02:24.921 INFO:teuthology.orchestra.run.smithi033.stderr: raise AssertionError("%s not raised" % excName) 2023-12-07T22:02:24.921 INFO:teuthology.orchestra.run.smithi033.stderr:AssertionError: ClientError not raised 2023-12-07T22:02:24.921 INFO:teuthology.orchestra.run.smithi033.stderr:-------------------- >> begin captured logging << -------------------- 2023-12-07T22:02:24.921 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event choose-service-name: calling handler 2023-12-07T22:02:24.921 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2023-12-07T22:02:24.921 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler ._handler at 0x7f6b64e56430> 2023-12-07T22:02:24.921 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event creating-client-class.s3: calling handler 2023-12-07T22:02:24.921 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.endpoint: DEBUG: Setting s3 timeout as (60, 60) 2023-12-07T22:02:24.921 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.client: DEBUG: Registering retry handlers for service: s3 2023-12-07T22:02:24.921 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2023-12-07T22:02:24.921 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.921 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.922 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2023-12-07T22:02:24.922 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2023-12-07T22:02:24.922 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.922 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2023-12-07T22:02:24.922 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2023-12-07T22:02:24.922 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-xxboxwb5lcqae2a-212', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.24.96 Python/3.8.10 Linux/5.4.0-167-generic Botocore/1.27.96'}, 'body': b'', 'url': 'http://smithi033.front.sepia.ceph.com:80/test-client.0-xxboxwb5lcqae2a-212', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-xxboxwb5lcqae2a-212'}}} 2023-12-07T22:02:24.922 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.922 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.922 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2023-12-07T22:02:24.922 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.922 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.utils: DEBUG: Using S3 path style addressing. 2023-12-07T22:02:24.922 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2023-12-07T22:02:24.922 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.auth: DEBUG: CanonicalRequest: 2023-12-07T22:02:24.922 INFO:teuthology.orchestra.run.smithi033.stderr:PUT 2023-12-07T22:02:24.923 INFO:teuthology.orchestra.run.smithi033.stderr:/test-client.0-xxboxwb5lcqae2a-212 2023-12-07T22:02:24.923 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:24.923 INFO:teuthology.orchestra.run.smithi033.stderr:host:smithi033.front.sepia.ceph.com 2023-12-07T22:02:24.923 INFO:teuthology.orchestra.run.smithi033.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-12-07T22:02:24.923 INFO:teuthology.orchestra.run.smithi033.stderr:x-amz-date:20231207T215927Z 2023-12-07T22:02:24.923 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:24.923 INFO:teuthology.orchestra.run.smithi033.stderr:host;x-amz-content-sha256;x-amz-date 2023-12-07T22:02:24.923 INFO:teuthology.orchestra.run.smithi033.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-12-07T22:02:24.923 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.auth: DEBUG: StringToSign: 2023-12-07T22:02:24.923 INFO:teuthology.orchestra.run.smithi033.stderr:AWS4-HMAC-SHA256 2023-12-07T22:02:24.923 INFO:teuthology.orchestra.run.smithi033.stderr:20231207T215927Z 2023-12-07T22:02:24.923 INFO:teuthology.orchestra.run.smithi033.stderr:20231207/us-east-1/s3/aws4_request 2023-12-07T22:02:24.923 INFO:teuthology.orchestra.run.smithi033.stderr:2e56bdf698eeb83e786fcdecf204e2d4e2ead0a0c1ae885d53a6f5ea887664c3 2023-12-07T22:02:24.923 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.auth: DEBUG: Signature: 2023-12-07T22:02:24.923 INFO:teuthology.orchestra.run.smithi033.stderr:142c9d9b43c5dc8bccc92dac67a31d6ba2f5da3c0f3fee362851523d288a571e 2023-12-07T22:02:24.924 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2023-12-07T22:02:24.924 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.endpoint: DEBUG: Sending http request: 2023-12-07T22:02:24.924 INFO:teuthology.orchestra.run.smithi033.stderr:urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): smithi033.front.sepia.ceph.com:80 2023-12-07T22:02:24.924 INFO:teuthology.orchestra.run.smithi033.stderr:urllib3.connectionpool: DEBUG: http://smithi033.front.sepia.ceph.com:80 "PUT /test-client.0-xxboxwb5lcqae2a-212 HTTP/1.1" 200 0 2023-12-07T22:02:24.924 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx00000356c1353bf224c60-006572403f-105f-default', 'Content-Length': '0', 'Date': 'Thu, 07 Dec 2023 21:59:27 GMT'} 2023-12-07T22:02:24.924 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.parsers: DEBUG: Response body: 2023-12-07T22:02:24.924 INFO:teuthology.orchestra.run.smithi033.stderr:b'' 2023-12-07T22:02:24.924 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2023-12-07T22:02:24.924 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.retryhandler: DEBUG: No retry needed. 2023-12-07T22:02:24.924 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.924 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2023-12-07T22:02:24.924 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.924 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.924 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-parameter-build.s3.CreateBucket: calling handler 2023-12-07T22:02:24.924 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2023-12-07T22:02:24.925 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.925 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2023-12-07T22:02:24.925 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-call.s3.CreateBucket: calling handler 2023-12-07T22:02:24.925 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.endpoint: DEBUG: Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/test-client.0-xxboxwb5lcqae2a-212', 'query_string': {}, 'method': 'PUT', 'headers': {'x-amz-acl': 'public-read', 'User-Agent': 'Boto3/1.24.96 Python/3.8.10 Linux/5.4.0-167-generic Botocore/1.27.96'}, 'body': b'', 'url': 'http://smithi033.front.sepia.ceph.com:80/test-client.0-xxboxwb5lcqae2a-212', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'test-client.0-xxboxwb5lcqae2a-212'}}} 2023-12-07T22:02:24.925 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.925 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.925 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event choose-signer.s3.CreateBucket: calling handler 2023-12-07T22:02:24.925 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event before-sign.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.925 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.auth: DEBUG: Calculating signature using v4 auth. 2023-12-07T22:02:24.925 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.auth: DEBUG: CanonicalRequest: 2023-12-07T22:02:24.925 INFO:teuthology.orchestra.run.smithi033.stderr:PUT 2023-12-07T22:02:24.925 INFO:teuthology.orchestra.run.smithi033.stderr:/test-client.0-xxboxwb5lcqae2a-212 2023-12-07T22:02:24.925 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:24.925 INFO:teuthology.orchestra.run.smithi033.stderr:host:smithi033.front.sepia.ceph.com 2023-12-07T22:02:24.926 INFO:teuthology.orchestra.run.smithi033.stderr:x-amz-acl:public-read 2023-12-07T22:02:24.926 INFO:teuthology.orchestra.run.smithi033.stderr:x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-12-07T22:02:24.926 INFO:teuthology.orchestra.run.smithi033.stderr:x-amz-date:20231207T215927Z 2023-12-07T22:02:24.926 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:24.926 INFO:teuthology.orchestra.run.smithi033.stderr:host;x-amz-acl;x-amz-content-sha256;x-amz-date 2023-12-07T22:02:24.926 INFO:teuthology.orchestra.run.smithi033.stderr:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-12-07T22:02:24.926 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.auth: DEBUG: StringToSign: 2023-12-07T22:02:24.926 INFO:teuthology.orchestra.run.smithi033.stderr:AWS4-HMAC-SHA256 2023-12-07T22:02:24.926 INFO:teuthology.orchestra.run.smithi033.stderr:20231207T215927Z 2023-12-07T22:02:24.926 INFO:teuthology.orchestra.run.smithi033.stderr:20231207/us-east-1/s3/aws4_request 2023-12-07T22:02:24.926 INFO:teuthology.orchestra.run.smithi033.stderr:f0562220e2754ef555a5b128d61767f3abd42be67b9775e899319fa92f163dfe 2023-12-07T22:02:24.926 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.auth: DEBUG: Signature: 2023-12-07T22:02:24.926 INFO:teuthology.orchestra.run.smithi033.stderr:265acd67c0a1421630e80a31df6265b8f20c03b5584b62b634e2003edd7e0314 2023-12-07T22:02:24.926 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event request-created.s3.CreateBucket: calling handler 2023-12-07T22:02:24.927 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.endpoint: DEBUG: Sending http request: 2023-12-07T22:02:24.927 INFO:teuthology.orchestra.run.smithi033.stderr:urllib3.connectionpool: DEBUG: http://smithi033.front.sepia.ceph.com:80 "PUT /test-client.0-xxboxwb5lcqae2a-212 HTTP/1.1" 200 0 2023-12-07T22:02:24.927 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.parsers: DEBUG: Response headers: {'x-amz-request-id': 'tx000003fba3632b5908868-006572403f-105f-default', 'Content-Length': '0', 'Date': 'Thu, 07 Dec 2023 21:59:27 GMT'} 2023-12-07T22:02:24.927 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.parsers: DEBUG: Response body: 2023-12-07T22:02:24.927 INFO:teuthology.orchestra.run.smithi033.stderr:b'' 2023-12-07T22:02:24.927 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler 2023-12-07T22:02:24.927 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.retryhandler: DEBUG: No retry needed. 2023-12-07T22:02:24.927 INFO:teuthology.orchestra.run.smithi033.stderr:botocore.hooks: DEBUG: Event needs-retry.s3.CreateBucket: calling handler > 2023-12-07T22:02:24.927 INFO:teuthology.orchestra.run.smithi033.stderr:--------------------- >> end captured logging << --------------------- 2023-12-07T22:02:24.927 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:24.927 INFO:teuthology.orchestra.run.smithi033.stderr:---------------------------------------------------------------------- 2023-12-07T22:02:24.927 INFO:teuthology.orchestra.run.smithi033.stderr:Ran 566 tests in 223.389s 2023-12-07T22:02:24.927 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:24.927 INFO:teuthology.orchestra.run.smithi033.stderr:FAILED (SKIP=76, failures=2) 2023-12-07T22:02:24.947 INFO:teuthology.orchestra.run.smithi033.stderr:Exception ignored in: .generate at 0x7f6b65767900> 2023-12-07T22:02:24.947 INFO:teuthology.orchestra.run.smithi033.stderr:RuntimeError: generator ignored GeneratorExit 2023-12-07T22:02:25.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T22:02:25.051 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/s3tests.py", line 441, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi033 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain'" 2023-12-07T22:02:25.052 INFO:tasks.s3tests:Cleaning up boto... 2023-12-07T22:02:25.052 DEBUG:teuthology.orchestra.run.smithi033:> rm /home/ubuntu/cephtest/boto.cfg 2023-12-07T22:02:25.057 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2023-12-07T22:02:25.087 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setuser ceph since I am not root 2023-12-07T22:02:25.087 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setgroup ceph since I am not root 2023-12-07T22:02:25.100 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.093+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.102 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.093+0000 7fcaf6d70c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.102 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.093+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.102 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.097+0000 7fcaf6d70c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.103 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.097+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.103 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.097+0000 7fcaf6d70c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.103 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.097+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.103 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.097+0000 7fcaf6d70c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T22:02:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.117+0000 7fcaf6d70c80 20 RGWRados::pool_iterate: got zonegroup_info.8b14f025-a9cb-4535-8f2a-c4795b716a58 2023-12-07T22:02:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.117+0000 7fcaf6d70c80 20 RGWRados::pool_iterate: got zone_info.5de1c49b-6b66-4f92-9f67-9eb1162319b3 2023-12-07T22:02:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.117+0000 7fcaf6d70c80 20 RGWRados::pool_iterate: got zone_names.default 2023-12-07T22:02:25.123 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.117+0000 7fcaf6d70c80 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-07T22:02:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.117+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.117+0000 7fcaf6d70c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T22:02:25.124 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.117+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.117+0000 7fcaf6d70c80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T22:02:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.117+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.117+0000 7fcaf6d70c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T22:02:25.125 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.117+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.126 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.117+0000 7fcaf6d70c80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T22:02:25.126 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.117+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.126 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.121+0000 7fcaf6d70c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.127 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.121+0000 7fcaf6d70c80 10 cannot find current period zonegroup using local zonegroup 2023-12-07T22:02:25.127 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.121+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.127 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.121+0000 7fcaf6d70c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.127 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.121+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.128 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.121+0000 7fcaf6d70c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T22:02:25.128 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.121+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.129 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.121+0000 7fcaf6d70c80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T22:02:25.129 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.121+0000 7fcaf6d70c80 20 zonegroup default 2023-12-07T22:02:25.129 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.121+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.129 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.121+0000 7fcaf6d70c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.129 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.121+0000 7fcaf6d70c80 10 Cannot find current period zone using local zone 2023-12-07T22:02:25.129 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.121+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.130 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.121+0000 7fcaf6d70c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.130 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.121+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.131 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.125+0000 7fcaf6d70c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T22:02:25.131 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.125+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.132 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.125+0000 7fcaf6d70c80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T22:02:25.132 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.125+0000 7fcaf6d70c80 20 zone default found 2023-12-07T22:02:25.132 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.125+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.132 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.125+0000 7fcaf6d70c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.132 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.125+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.133 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.125+0000 7fcaf6d70c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.133 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.125+0000 7fcaf6d70c80 20 started sync module instance, tier type = 2023-12-07T22:02:25.133 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.125+0000 7fcaf6d70c80 20 started zone id=5de1c49b-6b66-4f92-9f67-9eb1162319b3 (name=default) with tier type = 2023-12-07T22:02:25.167 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.161+0000 7fcaf6d70c80 20 add_watcher() i=0 2023-12-07T22:02:25.167 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.161+0000 7fcaf6d70c80 20 add_watcher() i=1 2023-12-07T22:02:25.167 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.161+0000 7fcaf6d70c80 20 add_watcher() i=2 2023-12-07T22:02:25.167 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.161+0000 7fcaf6d70c80 20 add_watcher() i=3 2023-12-07T22:02:25.167 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.161+0000 7fcaf6d70c80 20 add_watcher() i=4 2023-12-07T22:02:25.167 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.161+0000 7fcaf6d70c80 20 add_watcher() i=5 2023-12-07T22:02:25.167 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.161+0000 7fcaf6d70c80 20 add_watcher() i=6 2023-12-07T22:02:25.169 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.161+0000 7fcaf6d70c80 20 add_watcher() i=7 2023-12-07T22:02:25.169 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.161+0000 7fcaf6d70c80 2 all 8 watchers are set, enabling cache 2023-12-07T22:02:25.173 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.165+0000 7fcac67e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T22:02:25.173 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.165+0000 7fcaf6d70c80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-12-07T22:02:25.173 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.165+0000 7fcaf6d70c80 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-07T22:02:25.279 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.273+0000 7fca357fa700 20 reqs_thread_entry: start 2023-12-07T22:02:25.280 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.273+0000 7fcaf6d70c80 20 init_complete bucket index max shards: 11 2023-12-07T22:02:25.280 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.273+0000 7fca2effd700 20 reqs_thread_entry: start 2023-12-07T22:02:25.280 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.273+0000 7fcaf6d70c80 10 rgw notify: Started notification manager with: 1 workers 2023-12-07T22:02:25.280 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.273+0000 7fca2d7fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Dec 7 22:02:55 2023 2023-12-07T22:02:25.280 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:25.281 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.273+0000 7fca2cff9700 20 reqs_thread_entry: start 2023-12-07T22:02:25.281 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.273+0000 7fcaf6d70c80 20 get_system_obj_state: rctx=0x7fffddc3f778 obj=default.rgw.meta:users.uid:foo.client.0 state=0x5635e90c2020 s->prefetch_data=0 2023-12-07T22:02:25.281 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.273+0000 7fcaf6d70c80 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-12-07T22:02:25.281 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.273+0000 7fcaf6d70c80 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-12-07T22:02:25.281 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.273+0000 7fcaf6d70c80 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-12-07T22:02:25.281 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.273+0000 7fcaf6d70c80 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-07T22:02:25.281 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.273+0000 7fcaf6d70c80 20 get_system_obj_state: s->obj_tag was set empty 2023-12-07T22:02:25.281 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.273+0000 7fcaf6d70c80 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x11, cached=0x16) 2023-12-07T22:02:25.282 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.273+0000 7fcaf6d70c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.282 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.277+0000 7fcaf6d70c80 20 rados_obj.operate() r=0 bl.length=469 2023-12-07T22:02:25.282 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.277+0000 7fcaf6d70c80 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x11 2023-12-07T22:02:25.282 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.277+0000 7fcaf6d70c80 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-12-07T22:02:25.286 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.277+0000 7fcaf6d70c80 10 removing key index: JZGLQKOZONUJRVXRXMQY 2023-12-07T22:02:25.286 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.277+0000 7fcaf6d70c80 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.keys:JZGLQKOZONUJRVXRXMQY, ofs0, ns] 2023-12-07T22:02:25.287 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.281+0000 7fcaf299a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323857330 cookie 94789544470320 notifier 4233 bl.length()=170 2023-12-07T22:02:25.290 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.285+0000 7fcaf6d70c80 10 removing email index: foo.client.0+test@test.test 2023-12-07T22:02:25.290 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.285+0000 7fcaf6d70c80 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-12-07T22:02:25.291 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.285+0000 7fcaf299a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856244 cookie 94789544483472 notifier 4233 bl.length()=178 2023-12-07T22:02:25.294 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.289+0000 7fcaf6d70c80 10 removing user buckets index 2023-12-07T22:02:25.294 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.289+0000 7fcaf6d70c80 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2023-12-07T22:02:25.295 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.289+0000 7fcaf299a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856245 cookie 94789544467936 notifier 4233 bl.length()=169 2023-12-07T22:02:25.301 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.293+0000 7fcaf6d70c80 10 removing user index: foo.client.0 2023-12-07T22:02:25.301 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.293+0000 7fcaf6d70c80 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2023-12-07T22:02:25.301 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.293+0000 7fcaf6d70c80 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-12-07T22:02:25.302 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.293+0000 7fcaf299a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855652 cookie 94789544463008 notifier 4233 bl.length()=161 2023-12-07T22:02:25.309 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.301+0000 7fcaf6d70c80 20 remove_watcher() i=0 2023-12-07T22:02:25.309 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.301+0000 7fcaf6d70c80 2 removed watcher, disabling cache 2023-12-07T22:02:25.310 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.305+0000 7fcaf6d70c80 20 remove_watcher() i=1 2023-12-07T22:02:25.313 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.305+0000 7fcaf6d70c80 20 remove_watcher() i=2 2023-12-07T22:02:25.314 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.309+0000 7fcaf6d70c80 20 remove_watcher() i=3 2023-12-07T22:02:25.316 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.309+0000 7fcaf6d70c80 20 remove_watcher() i=4 2023-12-07T22:02:25.319 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.313+0000 7fcaf6d70c80 20 remove_watcher() i=5 2023-12-07T22:02:25.321 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.313+0000 7fcaf6d70c80 20 remove_watcher() i=6 2023-12-07T22:02:25.323 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.317+0000 7fcaf6d70c80 20 remove_watcher() i=7 2023-12-07T22:02:25.323 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.317+0000 7fcac67e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T22:02:25.332 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2023-12-07T22:02:25.411 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setuser ceph since I am not root 2023-12-07T22:02:25.411 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setgroup ceph since I am not root 2023-12-07T22:02:25.424 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.417+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.425 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.417+0000 7f9646871c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.425 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.417+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.426 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.417+0000 7f9646871c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.426 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.417+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.426 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.421+0000 7f9646871c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.426 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.421+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.427 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.421+0000 7f9646871c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T22:02:25.449 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.441+0000 7f9646871c80 20 RGWRados::pool_iterate: got zonegroup_info.8b14f025-a9cb-4535-8f2a-c4795b716a58 2023-12-07T22:02:25.449 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.441+0000 7f9646871c80 20 RGWRados::pool_iterate: got zone_info.5de1c49b-6b66-4f92-9f67-9eb1162319b3 2023-12-07T22:02:25.449 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.441+0000 7f9646871c80 20 RGWRados::pool_iterate: got zone_names.default 2023-12-07T22:02:25.449 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.441+0000 7f9646871c80 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-07T22:02:25.449 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.441+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.450 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.441+0000 7f9646871c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T22:02:25.450 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.441+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.451 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.445+0000 7f9646871c80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T22:02:25.451 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.445+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.452 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.445+0000 7f9646871c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T22:02:25.452 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.445+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.452 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.445+0000 7f9646871c80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T22:02:25.452 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.445+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.453 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.445+0000 7f9646871c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.453 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.445+0000 7f9646871c80 10 cannot find current period zonegroup using local zonegroup 2023-12-07T22:02:25.453 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.445+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.453 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.445+0000 7f9646871c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.453 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.445+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.454 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.449+0000 7f9646871c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T22:02:25.454 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.449+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.455 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.449+0000 7f9646871c80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T22:02:25.455 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.449+0000 7f9646871c80 20 zonegroup default 2023-12-07T22:02:25.455 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.449+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.456 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.449+0000 7f9646871c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.456 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.449+0000 7f9646871c80 10 Cannot find current period zone using local zone 2023-12-07T22:02:25.456 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.449+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.456 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.449+0000 7f9646871c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.456 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.449+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.457 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.449+0000 7f9646871c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T22:02:25.457 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.449+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.449+0000 7f9646871c80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T22:02:25.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.449+0000 7f9646871c80 20 zone default found 2023-12-07T22:02:25.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.449+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.453+0000 7f9646871c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.458 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.453+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.459 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.453+0000 7f9646871c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.459 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.453+0000 7f9646871c80 20 started sync module instance, tier type = 2023-12-07T22:02:25.459 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.453+0000 7f9646871c80 20 started zone id=5de1c49b-6b66-4f92-9f67-9eb1162319b3 (name=default) with tier type = 2023-12-07T22:02:25.486 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.477+0000 7f9646871c80 20 add_watcher() i=0 2023-12-07T22:02:25.486 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.477+0000 7f9646871c80 20 add_watcher() i=1 2023-12-07T22:02:25.486 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.477+0000 7f9646871c80 20 add_watcher() i=2 2023-12-07T22:02:25.486 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.477+0000 7f9646871c80 20 add_watcher() i=3 2023-12-07T22:02:25.486 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.477+0000 7f9646871c80 20 add_watcher() i=4 2023-12-07T22:02:25.486 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.477+0000 7f9646871c80 20 add_watcher() i=5 2023-12-07T22:02:25.486 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.477+0000 7f9646871c80 20 add_watcher() i=6 2023-12-07T22:02:25.487 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.481+0000 7f9646871c80 20 add_watcher() i=7 2023-12-07T22:02:25.488 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.481+0000 7f9646871c80 2 all 8 watchers are set, enabling cache 2023-12-07T22:02:25.491 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.485+0000 7f9614ff1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T22:02:25.491 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.485+0000 7f9646871c80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-12-07T22:02:25.491 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.485+0000 7f9646871c80 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-07T22:02:25.596 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.589+0000 7f95837fe700 20 reqs_thread_entry: start 2023-12-07T22:02:25.596 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.589+0000 7f95817fa700 20 reqs_thread_entry: start 2023-12-07T22:02:25.597 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.589+0000 7f9646871c80 20 init_complete bucket index max shards: 11 2023-12-07T22:02:25.597 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.589+0000 7f9646871c80 10 rgw notify: Started notification manager with: 1 workers 2023-12-07T22:02:25.597 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.589+0000 7f956f7fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Dec 7 22:02:55 2023 2023-12-07T22:02:25.597 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:25.597 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.589+0000 7f956effd700 20 reqs_thread_entry: start 2023-12-07T22:02:25.597 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.589+0000 7f9646871c80 20 get_system_obj_state: rctx=0x7ffc7845c138 obj=default.rgw.meta:users.uid:bar.client.0 state=0x561479804020 s->prefetch_data=0 2023-12-07T22:02:25.598 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.589+0000 7f9646871c80 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-12-07T22:02:25.598 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.589+0000 7f9646871c80 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-12-07T22:02:25.598 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.589+0000 7f9646871c80 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-12-07T22:02:25.598 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.589+0000 7f9646871c80 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-07T22:02:25.598 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.589+0000 7f9646871c80 20 get_system_obj_state: s->obj_tag was set empty 2023-12-07T22:02:25.598 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.589+0000 7f9646871c80 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x11, cached=0x16) 2023-12-07T22:02:25.598 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.589+0000 7f9646871c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.599 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.593+0000 7f9646871c80 20 rados_obj.operate() r=0 bl.length=469 2023-12-07T22:02:25.599 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.593+0000 7f9646871c80 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x11 2023-12-07T22:02:25.599 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.593+0000 7f9646871c80 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-12-07T22:02:25.600 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.593+0000 7f9646871c80 10 removing key index: WDEERDXDFEFVEGNXAPHG 2023-12-07T22:02:25.600 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.593+0000 7f9646871c80 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.keys:WDEERDXDFEFVEGNXAPHG, ofs0, ns] 2023-12-07T22:02:25.601 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.593+0000 7f9642c9c700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856246 cookie 94645939110688 notifier 4239 bl.length()=170 2023-12-07T22:02:25.605 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.597+0000 7f9646871c80 10 removing email index: bar.client.0+test@test.test 2023-12-07T22:02:25.605 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.597+0000 7f9646871c80 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-12-07T22:02:25.605 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.597+0000 7f9642c9c700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856247 cookie 94645939110688 notifier 4239 bl.length()=178 2023-12-07T22:02:25.609 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.601+0000 7f9646871c80 10 removing user buckets index 2023-12-07T22:02:25.609 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.601+0000 7f9646871c80 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0.buckets, ofs0, ns] 2023-12-07T22:02:25.609 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.601+0000 7f964249b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323857331 cookie 94645939114336 notifier 4239 bl.length()=169 2023-12-07T22:02:25.612 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.605+0000 7f9646871c80 10 removing user index: bar.client.0 2023-12-07T22:02:25.612 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.605+0000 7f9646871c80 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2023-12-07T22:02:25.613 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.605+0000 7f9646871c80 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-12-07T22:02:25.613 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.605+0000 7f964249b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856248 cookie 94645939110688 notifier 4239 bl.length()=161 2023-12-07T22:02:25.621 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.613+0000 7f9646871c80 20 remove_watcher() i=0 2023-12-07T22:02:25.621 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.613+0000 7f9646871c80 2 removed watcher, disabling cache 2023-12-07T22:02:25.623 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.617+0000 7f9646871c80 20 remove_watcher() i=1 2023-12-07T22:02:25.625 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.617+0000 7f9646871c80 20 remove_watcher() i=2 2023-12-07T22:02:25.629 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.621+0000 7f9646871c80 20 remove_watcher() i=3 2023-12-07T22:02:25.632 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.625+0000 7f9646871c80 20 remove_watcher() i=4 2023-12-07T22:02:25.634 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.629+0000 7f9646871c80 20 remove_watcher() i=5 2023-12-07T22:02:25.637 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.629+0000 7f9646871c80 20 remove_watcher() i=6 2023-12-07T22:02:25.639 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.633+0000 7f9646871c80 20 remove_watcher() i=7 2023-12-07T22:02:25.640 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.633+0000 7f9614ff1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T22:02:25.650 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2023-12-07T22:02:25.686 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setuser ceph since I am not root 2023-12-07T22:02:25.686 INFO:teuthology.orchestra.run.smithi033.stderr:ignoring --setgroup ceph since I am not root 2023-12-07T22:02:25.696 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.689+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.699 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.693+0000 7f5175497c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.700 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.693+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.700 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.693+0000 7f5175497c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.700 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.693+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.700 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.693+0000 7f5175497c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.700 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.693+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.701 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.693+0000 7f5175497c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T22:02:25.725 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.717+0000 7f5175497c80 20 RGWRados::pool_iterate: got zonegroup_info.8b14f025-a9cb-4535-8f2a-c4795b716a58 2023-12-07T22:02:25.725 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.717+0000 7f5175497c80 20 RGWRados::pool_iterate: got zone_info.5de1c49b-6b66-4f92-9f67-9eb1162319b3 2023-12-07T22:02:25.725 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.717+0000 7f5175497c80 20 RGWRados::pool_iterate: got zone_names.default 2023-12-07T22:02:25.725 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.717+0000 7f5175497c80 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-07T22:02:25.725 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.717+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.726 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.717+0000 7f5175497c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T22:02:25.726 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.717+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.726 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.717+0000 7f5175497c80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T22:02:25.726 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.717+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.727 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.721+0000 7f5175497c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T22:02:25.727 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.721+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.727 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.721+0000 7f5175497c80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T22:02:25.728 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.721+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.728 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.721+0000 7f5175497c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.728 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.721+0000 7f5175497c80 10 cannot find current period zonegroup using local zonegroup 2023-12-07T22:02:25.728 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.721+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.729 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.721+0000 7f5175497c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.729 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.721+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.730 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.721+0000 7f5175497c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T22:02:25.730 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.721+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.731 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.725+0000 7f5175497c80 20 rados_obj.operate() r=0 bl.length=358 2023-12-07T22:02:25.731 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.725+0000 7f5175497c80 20 zonegroup default 2023-12-07T22:02:25.731 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.725+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.731 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.725+0000 7f5175497c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.731 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.725+0000 7f5175497c80 10 Cannot find current period zone using local zone 2023-12-07T22:02:25.731 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.725+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.732 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.725+0000 7f5175497c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.732 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.725+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.732 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.725+0000 7f5175497c80 20 rados_obj.operate() r=0 bl.length=46 2023-12-07T22:02:25.732 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.725+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.733 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.725+0000 7f5175497c80 20 rados_obj.operate() r=0 bl.length=873 2023-12-07T22:02:25.733 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.725+0000 7f5175497c80 20 zone default found 2023-12-07T22:02:25.733 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.725+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.734 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.725+0000 7f5175497c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.734 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.725+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.734 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.729+0000 7f5175497c80 20 rados_obj.operate() r=-2 bl.length=0 2023-12-07T22:02:25.734 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.729+0000 7f5175497c80 20 started sync module instance, tier type = 2023-12-07T22:02:25.734 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.729+0000 7f5175497c80 20 started zone id=5de1c49b-6b66-4f92-9f67-9eb1162319b3 (name=default) with tier type = 2023-12-07T22:02:25.755 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.749+0000 7f5175497c80 20 add_watcher() i=0 2023-12-07T22:02:25.755 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.749+0000 7f5175497c80 20 add_watcher() i=1 2023-12-07T22:02:25.755 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.749+0000 7f5175497c80 20 add_watcher() i=2 2023-12-07T22:02:25.755 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.749+0000 7f5175497c80 20 add_watcher() i=3 2023-12-07T22:02:25.755 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.749+0000 7f5175497c80 20 add_watcher() i=4 2023-12-07T22:02:25.755 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.749+0000 7f5175497c80 20 add_watcher() i=5 2023-12-07T22:02:25.756 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.749+0000 7f5175497c80 20 add_watcher() i=6 2023-12-07T22:02:25.756 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.749+0000 7f5175497c80 20 add_watcher() i=7 2023-12-07T22:02:25.756 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.749+0000 7f5175497c80 2 all 8 watchers are set, enabling cache 2023-12-07T22:02:25.760 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.753+0000 7f5175497c80 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-12-07T22:02:25.760 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.753+0000 7f5175497c80 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-07T22:02:25.760 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.753+0000 7f5144fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T22:02:25.857 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.849+0000 7f50b37fe700 20 reqs_thread_entry: start 2023-12-07T22:02:25.858 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.849+0000 7f50b17fa700 20 reqs_thread_entry: start 2023-12-07T22:02:25.858 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.849+0000 7f5175497c80 20 init_complete bucket index max shards: 11 2023-12-07T22:02:25.858 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.849+0000 7f5175497c80 10 rgw notify: Started notification manager with: 1 workers 2023-12-07T22:02:25.858 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.849+0000 7f509f7fe700 20 rgw notify: INFO: next queues processing will happen at: Thu Dec 7 22:02:56 2023 2023-12-07T22:02:25.858 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:25.858 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.849+0000 7f5175497c80 20 get_system_obj_state: rctx=0x7ffd4a86dca8 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55d31f736020 s->prefetch_data=0 2023-12-07T22:02:25.858 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.849+0000 7f5175497c80 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-12-07T22:02:25.858 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.849+0000 7f509effd700 20 reqs_thread_entry: start 2023-12-07T22:02:25.859 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.853+0000 7f5175497c80 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-12-07T22:02:25.859 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.853+0000 7f5175497c80 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-12-07T22:02:25.859 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.853+0000 7f5175497c80 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-07T22:02:25.859 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.853+0000 7f5175497c80 10 updating xattr: name=user.rgw.user-policy bl.length()=118 2023-12-07T22:02:25.859 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.853+0000 7f5175497c80 20 get_system_obj_state: s->obj_tag was set empty 2023-12-07T22:02:25.859 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.853+0000 7f5175497c80 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x11, cached=0x16) 2023-12-07T22:02:25.859 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.853+0000 7f5175497c80 20 rados->read ofs=0 len=0 2023-12-07T22:02:25.859 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.853+0000 7f5175497c80 20 rados_obj.operate() r=0 bl.length=528 2023-12-07T22:02:25.860 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.853+0000 7f5175497c80 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x11 2023-12-07T22:02:25.860 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.853+0000 7f5175497c80 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-12-07T22:02:25.860 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.853+0000 7f5175497c80 10 removing key index: JYGIAADVMWOHZERHZHDD 2023-12-07T22:02:25.861 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.853+0000 7f5175497c80 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.keys:JYGIAADVMWOHZERHZHDD, ofs0, ns] 2023-12-07T22:02:25.861 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.853+0000 7f51710c1700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856249 cookie 94365255443200 notifier 4245 bl.length()=170 2023-12-07T22:02:25.864 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.857+0000 7f5175497c80 10 removing email index: testx$tenanteduser.client.0+test@test.test 2023-12-07T22:02:25.865 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.857+0000 7f5175497c80 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-12-07T22:02:25.865 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.857+0000 7f51708c0700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855653 cookie 94365255464672 notifier 4245 bl.length()=193 2023-12-07T22:02:25.869 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.861+0000 7f5175497c80 10 removing user buckets index 2023-12-07T22:02:25.869 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.861+0000 7f5175497c80 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0.buckets, ofs0, ns] 2023-12-07T22:02:25.869 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.861+0000 7f51708c0700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323856250 cookie 94365255455696 notifier 4245 bl.length()=184 2023-12-07T22:02:25.873 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.865+0000 7f5175497c80 10 removing user index: testx$tenanteduser.client.0 2023-12-07T22:02:25.873 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.865+0000 7f5175497c80 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2023-12-07T22:02:25.873 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.865+0000 7f5175497c80 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-12-07T22:02:25.873 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.865+0000 7f51710c1700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 150323855654 cookie 94365255446720 notifier 4245 bl.length()=176 2023-12-07T22:02:25.880 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.873+0000 7f5175497c80 20 remove_watcher() i=0 2023-12-07T22:02:25.880 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.873+0000 7f5175497c80 2 removed watcher, disabling cache 2023-12-07T22:02:25.883 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.877+0000 7f5175497c80 20 remove_watcher() i=1 2023-12-07T22:02:25.885 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.877+0000 7f5175497c80 20 remove_watcher() i=2 2023-12-07T22:02:25.889 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.881+0000 7f5175497c80 20 remove_watcher() i=3 2023-12-07T22:02:25.890 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.881+0000 7f5175497c80 20 remove_watcher() i=4 2023-12-07T22:02:25.892 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.885+0000 7f5175497c80 20 remove_watcher() i=5 2023-12-07T22:02:25.897 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.889+0000 7f5175497c80 20 remove_watcher() i=6 2023-12-07T22:02:25.899 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.893+0000 7f5175497c80 20 remove_watcher() i=7 2023-12-07T22:02:25.899 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-07T22:02:25.893+0000 7f5144fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-07T22:02:25.909 INFO:tasks.s3tests:Removing s3-tests... 2023-12-07T22:02:25.909 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf /home/ubuntu/cephtest/s3-tests 2023-12-07T22:02:26.030 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/s3tests.py", line 639, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/s3tests.py", line 59, in download yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/s3tests.py", line 441, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi033 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain'" 2023-12-07T22:02:26.103 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7a7acadc3f1a4d69a5223f2f16d6e2bf Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/s3tests.py", line 639, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/s3tests.py", line 59, in download yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_451ceae11b5d4caee05e94372fe658621eefc0b1/qa/tasks/s3tests.py", line 441, in run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (s3 tests against rgw) on smithi033 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain'" 2023-12-07T22:02:26.106 DEBUG:teuthology.run_tasks:Unwinding manager s3tests 2023-12-07T22:02:26.116 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-12-07T22:02:26.127 DEBUG:tasks.rgw.client.0:waiting for process to exit 2023-12-07T22:02:26.127 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T22:02:26.132 INFO:tasks.rgw.client.0.smithi033.stdout:2023-12-07T22:02:26.121+0000 7f1f5effd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_kmip_addr 172.21.15.33:5696 --foreground (PID: 32443) UID: 0 2023-12-07T22:02:26.132 INFO:tasks.rgw.client.0.smithi033.stdout:2023-12-07T22:02:26.121+0000 7f1f6f6e0cc0 -1 shutting down 2023-12-07T22:02:26.759 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2023-12-07T22:02:32.228 INFO:tasks.rgw.client.0:Stopped 2023-12-07T22:02:32.228 DEBUG:teuthology.orchestra.run.smithi033:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2023-12-07T22:02:32.236 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /etc/ceph/vault-root-token 2023-12-07T22:02:32.251 DEBUG:teuthology.run_tasks:Unwinding manager pykmip 2023-12-07T22:02:32.260 INFO:tasks.pykmip:Stopping PyKMIP instance 2023-12-07T22:02:32.260 DEBUG:tasks.pykmip.client.0:waiting for process to exit 2023-12-07T22:02:32.260 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T22:02:32.263 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~6s 2023-12-07T22:02:36.371 INFO:tasks.pykmip.client.0.smithi033.stderr:/home/ubuntu/cephtest/pykmip/kmip/services/server/crypto/engine.py:54: CryptographyDeprecationWarning: Blowfish has been deprecated 2023-12-07T22:02:36.371 INFO:tasks.pykmip.client.0.smithi033.stderr: enums.CryptographicAlgorithm.BLOWFISH: algorithms.Blowfish, 2023-12-07T22:02:36.371 INFO:tasks.pykmip.client.0.smithi033.stderr:/home/ubuntu/cephtest/pykmip/kmip/services/server/crypto/engine.py:56: CryptographyDeprecationWarning: CAST5 has been deprecated 2023-12-07T22:02:36.371 INFO:tasks.pykmip.client.0.smithi033.stderr: enums.CryptographicAlgorithm.CAST5: algorithms.CAST5, 2023-12-07T22:02:36.371 INFO:tasks.pykmip.client.0.smithi033.stderr:/home/ubuntu/cephtest/pykmip/kmip/services/server/crypto/engine.py:57: CryptographyDeprecationWarning: IDEA has been deprecated 2023-12-07T22:02:36.372 INFO:tasks.pykmip.client.0.smithi033.stderr: enums.CryptographicAlgorithm.IDEA: algorithms.IDEA, 2023-12-07T22:02:36.372 INFO:tasks.pykmip.client.0.smithi033.stderr:/home/ubuntu/cephtest/pykmip/kmip/services/server/engine.py:438: SAWarning: TypeDecorator EnumType() will not produce a cache key because the ``cache_ok`` attribute is not set to True. This can have significant performance implications including some performance degradations in comparison to prior SQLAlchemy versions. Set this attribute to True if this type object's state is safe to use in a cache key, or False to disable this warning. (Background on this warning at: https://sqlalche.me/e/20/cprf) 2023-12-07T22:02:36.372 INFO:tasks.pykmip.client.0.smithi033.stderr: object_type = self._data_session.query( 2023-12-07T22:02:36.493 INFO:tasks.pykmip.client.0:Stopped 2023-12-07T22:02:36.494 INFO:tasks.pykmip:Removing system dependencies of PyKMIP... 2023-12-07T22:02:36.494 DEBUG:teuthology.orchestra.run.smithi033:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y purge libffi-dev 2023-12-07T22:02:36.612 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-07T22:02:36.822 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-12-07T22:02:36.822 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-12-07T22:02:37.026 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-12-07T22:02:37.027 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-07T22:02:37.027 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-07T22:02:37.055 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages will be REMOVED: 2023-12-07T22:02:37.056 INFO:teuthology.orchestra.run.smithi033.stdout: libffi-dev* 2023-12-07T22:02:37.441 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 0 newly installed, 1 to remove and 255 not upgraded. 2023-12-07T22:02:37.441 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 312 kB disk space will be freed. 2023-12-07T22:02:37.564 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 149615 files and directories currently installed.) 2023-12-07T22:02:37.569 INFO:teuthology.orchestra.run.smithi033.stdout:Removing libffi-dev:amd64 (3.3-4) ... 2023-12-07T22:02:37.766 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2023-12-07T22:02:38.117 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2023-12-07T22:02:38.445 INFO:teuthology.orchestra.run.smithi033.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-12-07T22:02:40.100 DEBUG:teuthology.orchestra.run.smithi033:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y purge libsqlite3-dev 2023-12-07T22:02:40.176 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-07T22:02:40.389 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-12-07T22:02:40.390 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-12-07T22:02:40.595 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-12-07T22:02:40.596 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-07T22:02:40.596 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-07T22:02:40.626 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages will be REMOVED: 2023-12-07T22:02:40.627 INFO:teuthology.orchestra.run.smithi033.stdout: libsqlite3-dev* 2023-12-07T22:02:41.008 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 0 newly installed, 1 to remove and 255 not upgraded. 2023-12-07T22:02:41.008 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 2,372 kB disk space will be freed. 2023-12-07T22:02:41.125 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 149581 files and directories currently installed.) 2023-12-07T22:02:41.130 INFO:teuthology.orchestra.run.smithi033.stdout:Removing libsqlite3-dev:amd64 (3.31.1-4ubuntu0.5) ... 2023-12-07T22:02:41.983 DEBUG:teuthology.orchestra.run.smithi033:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y purge libssl-dev 2023-12-07T22:02:42.059 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-07T22:02:42.269 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-12-07T22:02:42.270 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-12-07T22:02:42.467 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-12-07T22:02:42.468 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-07T22:02:42.468 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-07T22:02:42.497 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages will be REMOVED: 2023-12-07T22:02:42.497 INFO:teuthology.orchestra.run.smithi033.stdout: libssl-dev* 2023-12-07T22:02:42.908 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 0 newly installed, 1 to remove and 255 not upgraded. 2023-12-07T22:02:42.909 INFO:teuthology.orchestra.run.smithi033.stdout:After this operation, 8,010 kB disk space will be freed. 2023-12-07T22:02:43.036 INFO:teuthology.orchestra.run.smithi033.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 149572 files and directories currently installed.) 2023-12-07T22:02:43.041 INFO:teuthology.orchestra.run.smithi033.stdout:Removing libssl-dev:amd64 (1.1.1f-1ubuntu2.20) ... 2023-12-07T22:02:43.911 DEBUG:teuthology.orchestra.run.smithi033:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y purge python-dev 2023-12-07T22:02:43.986 INFO:teuthology.orchestra.run.smithi033.stdout:Reading package lists... 2023-12-07T22:02:44.180 INFO:teuthology.orchestra.run.smithi033.stdout:Building dependency tree... 2023-12-07T22:02:44.181 INFO:teuthology.orchestra.run.smithi033.stdout:Reading state information... 2023-12-07T22:02:44.360 INFO:teuthology.orchestra.run.smithi033.stdout:The following packages were automatically installed and are no longer required: 2023-12-07T22:02:44.361 INFO:teuthology.orchestra.run.smithi033.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-12-07T22:02:44.361 INFO:teuthology.orchestra.run.smithi033.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-07T22:02:44.410 INFO:teuthology.orchestra.run.smithi033.stdout:0 upgraded, 0 newly installed, 0 to remove and 255 not upgraded. 2023-12-07T22:02:44.412 INFO:tasks.pykmip:Removing pykmip... 2023-12-07T22:02:44.412 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf /home/ubuntu/cephtest/pykmip 2023-12-07T22:02:44.480 DEBUG:teuthology.run_tasks:Unwinding manager openssl_keys 2023-12-07T22:02:44.491 DEBUG:teuthology.orchestra.run.smithi033:> rm -f /home/ubuntu/cephtest/ca/kmiproot.crt /home/ubuntu/cephtest/ca/kmiproot.key 2023-12-07T22:02:44.497 DEBUG:teuthology.orchestra.run.smithi033:> rmdir --ignore-fail-on-non-empty /home/ubuntu/cephtest/ca 2023-12-07T22:02:44.519 DEBUG:teuthology.orchestra.run.smithi033:> rm -f /home/ubuntu/cephtest/ca/kmip-client.crt /home/ubuntu/cephtest/ca/kmip-client.key 2023-12-07T22:02:44.524 DEBUG:teuthology.orchestra.run.smithi033:> rmdir --ignore-fail-on-non-empty /home/ubuntu/cephtest/ca 2023-12-07T22:02:44.572 DEBUG:teuthology.orchestra.run.smithi033:> rm -f /home/ubuntu/cephtest/ca/kmip-server.crt /home/ubuntu/cephtest/ca/kmip-server.key 2023-12-07T22:02:44.578 DEBUG:teuthology.orchestra.run.smithi033:> rmdir --ignore-fail-on-non-empty /home/ubuntu/cephtest/ca 2023-12-07T22:02:44.585 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-07T22:02:44.596 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-12-07T22:02:44.597 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-07T22:02:44.915 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-07T22:02:44.916 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":35,"fsid":"d80f57a9-566f-4802-89c6-ac2d96473541","created":"2023-12-07T21:55:44.914448+0000","modified":"2023-12-07T21:58:50.005063+0000","last_up_change":"2023-12-07T21:55:51.902623+0000","last_in_change":"2023-12-07T21:55:45.771064+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-12-07T21:55:51.973567+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-07T21:55:53.407039+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2023-12-07T21:56:43.550312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2023-12-07T21:56:45.306137+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2023-12-07T21:56:47.341139+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2023-12-07T21:56:49.105198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2023-12-07T21:56:51.066481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2023-12-07T21:56:53.227828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"default.rgw.otp","create_time":"2023-12-07T21:57:03.573410+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"default.rgw.buckets.non-ec","create_time":"2023-12-07T21:58:42.350990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"dc992a6a-05d7-42e3-b943-34ea323f923c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":29685},{"type":"v1","addr":"172.21.15.33:6817","nonce":29685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":29685},{"type":"v1","addr":"172.21.15.33:6819","nonce":29685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":29685},{"type":"v1","addr":"172.21.15.33:6823","nonce":29685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":29685},{"type":"v1","addr":"172.21.15.33:6821","nonce":29685}]},"public_addr":"172.21.15.33:6817/29685","cluster_addr":"172.21.15.33:6819/29685","heartbeat_back_addr":"172.21.15.33:6823/29685","heartbeat_front_addr":"172.21.15.33:6821/29685","state":["exists","up"]},{"osd":1,"uuid":"7c55af6b-1aeb-49a6-94bd-3faccf2d6445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":29683},{"type":"v1","addr":"172.21.15.33:6801","nonce":29683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":29683},{"type":"v1","addr":"172.21.15.33:6803","nonce":29683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":29683},{"type":"v1","addr":"172.21.15.33:6807","nonce":29683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":29683},{"type":"v1","addr":"172.21.15.33:6805","nonce":29683}]},"public_addr":"172.21.15.33:6801/29683","cluster_addr":"172.21.15.33:6803/29683","heartbeat_back_addr":"172.21.15.33:6807/29683","heartbeat_front_addr":"172.21.15.33:6805/29683","state":["exists","up"]},{"osd":2,"uuid":"5c54b971-af63-4b5e-898a-ee8072a7524b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":29684},{"type":"v1","addr":"172.21.15.33:6809","nonce":29684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":29684},{"type":"v1","addr":"172.21.15.33:6811","nonce":29684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":29684},{"type":"v1","addr":"172.21.15.33:6815","nonce":29684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":29684},{"type":"v1","addr":"172.21.15.33:6813","nonce":29684}]},"public_addr":"172.21.15.33:6809/29684","cluster_addr":"172.21.15.33:6811/29684","heartbeat_back_addr":"172.21.15.33:6815/29684","heartbeat_front_addr":"172.21.15.33:6813/29684","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T21:55:49.936627+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T21:55:49.941640+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-12-07T21:55:50.195038+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":0,"to":1}]},{"pgid":"3.f","mappings":[{"from":0,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":0}]},{"pgid":"4.15","mappings":[{"from":1,"to":2}]},{"pgid":"4.2a","mappings":[{"from":1,"to":2}]},{"pgid":"4.2b","mappings":[{"from":1,"to":2}]},{"pgid":"4.3b","mappings":[{"from":1,"to":2}]},{"pgid":"10.12","mappings":[{"from":1,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-07T22:02:44.929 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-07T22:02:45.230 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-07T22:02:45.230 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-07T22:02:45.230 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-07T22:02:45.231 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T22:02:45.308 INFO:tasks.ceph.osd.0:Stopped 2023-12-07T22:02:45.308 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-07T22:02:45.308 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T22:02:45.375 INFO:tasks.ceph.osd.1:Stopped 2023-12-07T22:02:45.375 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-07T22:02:45.375 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T22:02:45.475 INFO:tasks.ceph.osd.2:Stopped 2023-12-07T22:02:45.476 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-07T22:02:45.476 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-07T22:02:45.476 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T22:02:45.496 INFO:tasks.ceph.mgr.x:Stopped 2023-12-07T22:02:45.496 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-07T22:02:45.497 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-07T22:02:45.497 INFO:teuthology.orchestra.run:waiting for 300 2023-12-07T22:02:45.517 INFO:tasks.ceph.mon.a:Stopped 2023-12-07T22:02:45.518 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-07T22:02:45.518 DEBUG:teuthology.orchestra.run.smithi033:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-12-07T22:02:45.536 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-07T22:02:45.537 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-07T22:02:46.340 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-07T22:02:46.340 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-07T22:02:46.597 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-07T22:02:46.599 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-07T22:02:46.804 INFO:tasks.ceph:Archiving mon data... 2023-12-07T22:02:46.805 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482401/data/mon.a.tgz 2023-12-07T22:02:46.805 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2023-12-07T22:02:46.810 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.GVc8PPdbYQ 2023-12-07T22:02:46.810 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.GVc8PPdbYQ 2023-12-07T22:02:47.058 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.GVc8PPdbYQ 2023-12-07T22:02:47.125 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.GVc8PPdbYQ is 1MB 2023-12-07T22:02:47.201 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.GVc8PPdbYQ 2023-12-07T22:02:47.208 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-07T22:02:47.208 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-07T22:02:47.224 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-12-07T22:02:47.225 DEBUG:teuthology.orchestra.run.smithi033:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-12-07T22:02:47.244 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-12-07T22:02:47.250 INFO:tasks.ceph:Archiving crash dumps... 2023-12-07T22:02:47.251 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482401/remote/smithi033/crash 2023-12-07T22:02:47.252 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-07T22:02:47.265 INFO:tasks.ceph:Compressing logs... 2023-12-07T22:02:47.265 DEBUG:teuthology.orchestra.run.smithi033:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-07T22:02:47.335 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31553.log 2023-12-07T22:02:47.335 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35195.log 2023-12-07T22:02:47.336 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31553.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-07T22:02:47.336 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31553.log.gz 2023-12-07T22:02:47.337 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26988.log 2023-12-07T22:02:47.337 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35195.log.gz 2023-12-07T22:02:47.337 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32317.log 2023-12-07T22:02:47.338 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph.tmp-client.admin.26988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.33245.log 2023-12-07T22:02:47.338 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26988.log.gz 2023-12-07T22:02:47.338 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35220.log 2023-12-07T22:02:47.339 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32317.log.gz 2023-12-07T22:02:47.339 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31526.log 2023-12-07T22:02:47.340 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.0.33245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.34526.log /var/log/ceph/ceph-client.admin.35220.log: 2023-12-07T22:02:47.340 INFO:teuthology.orchestra.run.smithi033.stderr: 86.6% 0.0% -- replaced with /var/log/ceph/ceph-client.0.33245.log.gz -- replaced with /var/log/ceph/ceph-client.admin.35220.log.gz 2023-12-07T22:02:47.341 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:47.341 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29620.log 2023-12-07T22:02:47.341 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31526.log.gz 2023-12-07T22:02:47.342 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.33399.log 2023-12-07T22:02:47.342 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.0.34526.log: /var/log/ceph/ceph-client.admin.29620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29495.log 2023-12-07T22:02:47.342 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29620.log.gz 2023-12-07T22:02:47.342 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.0.33399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.33476.log 2023-12-07T22:02:47.342 INFO:teuthology.orchestra.run.smithi033.stderr: 83.4% -- replaced with /var/log/ceph/ceph-client.0.34526.log.gz 2023-12-07T22:02:47.344 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- 86.6%/var/log/ceph/ceph-mgr.x.log 2023-12-07T22:02:47.344 INFO:teuthology.orchestra.run.smithi033.stderr: -- replaced with /var/log/ceph/ceph-client.0.33399.log.gz 2023-12-07T22:02:47.344 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29495.log.gz 2023-12-07T22:02:47.344 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.0.33476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32398.log 2023-12-07T22:02:47.344 INFO:teuthology.orchestra.run.smithi033.stderr: 83.7%/var/log/ceph/ceph-mgr.x.log: -- replaced with /var/log/ceph/ceph-client.0.33476.log.gz 2023-12-07T22:02:47.344 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-07T22:02:47.345 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29645.log 2023-12-07T22:02:47.345 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32398.log.gz 2023-12-07T22:02:47.345 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/rgw.ceph.client.0.log 2023-12-07T22:02:47.346 INFO:teuthology.orchestra.run.smithi033.stderr: 88.3%/var/log/ceph/ceph-client.admin.29645.log: -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-07T22:02:47.346 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29645.log.gz 2023-12-07T22:02:47.346 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.33166.log 2023-12-07T22:02:47.351 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/rgw.ceph.client.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.33322.log 2023-12-07T22:02:47.351 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.0.33166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31391.log 2023-12-07T22:02:47.352 INFO:teuthology.orchestra.run.smithi033.stderr: 83.5% -- replaced with /var/log/ceph/ceph-client.0.33166.log.gz 2023-12-07T22:02:47.352 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32369.log 2023-12-07T22:02:47.353 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.0.33322.log: /var/log/ceph/ceph-client.admin.31391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32344.log 2023-12-07T22:02:47.353 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31391.log.gz 2023-12-07T22:02:47.353 INFO:teuthology.orchestra.run.smithi033.stderr: 83.4% -- replaced with /var/log/ceph/ceph-client.0.33322.log.gz 2023-12-07T22:02:47.354 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-07T22:02:47.354 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32369.log.gz 2023-12-07T22:02:47.354 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31283.log 2023-12-07T22:02:47.354 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32344.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32344.log.gz 2023-12-07T22:02:47.354 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.33089.log 2023-12-07T22:02:47.355 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31256.log 2023-12-07T22:02:47.355 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31337.log 2023-12-07T22:02:47.356 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31283.log.gz 2023-12-07T22:02:47.356 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.0.33089.log: /var/log/ceph/ceph-client.admin.31256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31256.log.gz 2023-12-07T22:02:47.356 INFO:teuthology.orchestra.run.smithi033.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.0.33089.log.gz 2023-12-07T22:02:47.358 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31418.log 2023-12-07T22:02:47.360 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31229.log 2023-12-07T22:02:47.360 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31337.log.gz 2023-12-07T22:02:47.360 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.34603.log 2023-12-07T22:02:47.360 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31418.log.gz 2023-12-07T22:02:47.361 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31580.log 2023-12-07T22:02:47.361 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31229.log.gz 2023-12-07T22:02:47.361 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.0.34603.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-07T22:02:47.363 INFO:teuthology.orchestra.run.smithi033.stderr: 83.4% -- replaced with /var/log/ceph/ceph-client.0.34603.log.gz 2023-12-07T22:02:47.363 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.34449.log 2023-12-07T22:02:47.363 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31580.log.gz 2023-12-07T22:02:47.363 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-07T22:02:47.363 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.0.34449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31310.log 2023-12-07T22:02:47.363 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.log: 83.5% -- replaced with /var/log/ceph/ceph-client.0.34449.log.gz 2023-12-07T22:02:47.363 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31203.log 2023-12-07T22:02:47.364 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31310.log.gz 2023-12-07T22:02:47.364 INFO:teuthology.orchestra.run.smithi033.stderr: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-07T22:02:47.377 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31445.log 2023-12-07T22:02:47.377 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31499.log 2023-12-07T22:02:47.377 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31203.log.gz/var/log/ceph/ceph-client.admin.31445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31472.log 2023-12-07T22:02:47.377 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31445.log.gz 2023-12-07T22:02:47.377 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29775.log 2023-12-07T22:02:47.378 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31499.log: 2023-12-07T22:02:47.378 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31364.log 2023-12-07T22:02:47.378 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0%/var/log/ceph/ceph-client.admin.31472.log: -- replaced with /var/log/ceph/ceph-client.admin.31499.log.gz 2023-12-07T22:02:47.378 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31472.log.gz 2023-12-07T22:02:47.378 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31607.log 2023-12-07T22:02:47.378 INFO:teuthology.orchestra.run.smithi033.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29775.log.gz 2023-12-07T22:02:47.378 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31364.log.gz 2023-12-07T22:02:47.378 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29681.log 2023-12-07T22:02:47.380 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29593.log 2023-12-07T22:02:47.380 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31607.log.gz 2023-12-07T22:02:47.380 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29681.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-07T22:02:47.380 INFO:teuthology.orchestra.run.smithi033.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29681.log.gz 2023-12-07T22:02:47.380 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.29593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29593.log.gz 2023-12-07T22:02:47.381 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.2.log: 93.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-07T22:02:47.491 INFO:teuthology.orchestra.run.smithi033.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-07T22:02:48.588 INFO:teuthology.orchestra.run.smithi033.stderr: 93.9% -- replaced with /var/log/ceph/rgw.ceph.client.0.log.gz 2023-12-07T22:02:52.545 INFO:teuthology.orchestra.run.smithi033.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-07T22:02:53.418 INFO:teuthology.orchestra.run.smithi033.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-07T22:02:54.437 INFO:teuthology.orchestra.run.smithi033.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-07T22:02:54.438 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-07T22:02:54.438 INFO:teuthology.orchestra.run.smithi033.stderr:real 0m7.124s 2023-12-07T22:02:54.439 INFO:teuthology.orchestra.run.smithi033.stderr:user 0m19.117s 2023-12-07T22:02:54.439 INFO:teuthology.orchestra.run.smithi033.stderr:sys 0m0.710s 2023-12-07T22:02:54.440 INFO:tasks.ceph:Archiving logs... 2023-12-07T22:02:54.440 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482401/remote/smithi033/log 2023-12-07T22:02:54.441 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-07T22:02:57.125 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-07T22:02:57.139 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-07T22:02:57.140 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-07T22:02:57.198 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-07T22:02:57.208 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-07T22:02:57.208 DEBUG:teuthology.orchestra.run.smithi033:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-07T22:02:57.228 INFO:teuthology.orchestra.run.smithi033.stdout: remote refid st t when poll reach delay offset jitter 2023-12-07T22:02:57.228 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================== 2023-12-07T22:02:57.228 INFO:teuthology.orchestra.run.smithi033.stdout:*hv01.front.sepi 44.4.53.2 2 u 26 64 377 0.066 -1.151 1.214 2023-12-07T22:02:57.229 INFO:teuthology.orchestra.run.smithi033.stdout:+hv02.front.sepi 63.231.80.2 3 u 25 64 377 0.093 2.147 1.041 2023-12-07T22:02:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: hv03.front.sepi 74.6.168.72 3 u 24 64 377 0.083 3.657 1.766 2023-12-07T22:02:57.229 INFO:teuthology.orchestra.run.smithi033.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-07T22:02:57.230 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-07T22:02:57.242 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-07T22:02:57.243 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-07T22:02:57.289 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-07T22:02:57.304 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-07T22:02:57.318 INFO:teuthology.task.internal:Duration was 944.569888 seconds 2023-12-07T22:02:57.319 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-07T22:02:57.369 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-07T22:02:57.369 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-07T22:02:57.426 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-07T22:02:57.426 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi033.front.sepia.ceph.com 2023-12-07T22:02:57.426 DEBUG:teuthology.orchestra.run.smithi033:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-07T22:02:57.438 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-07T22:02:57.438 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-07T22:02:57.514 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-07T22:02:57.515 DEBUG:teuthology.orchestra.run.smithi033:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-07T22:02:57.587 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-07T22:02:57.600 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-07T22:02:57.600 DEBUG:teuthology.orchestra.run.smithi033:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-07T22:02:57.618 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-07T22:02:57.631 DEBUG:teuthology.orchestra.run.smithi033:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-07T22:02:57.671 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = core 2023-12-07T22:02:57.689 DEBUG:teuthology.orchestra.run.smithi033:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-07T22:02:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-07T22:02:57.695 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-07T22:02:57.708 INFO:teuthology.task.internal:Transferring archived files... 2023-12-07T22:02:57.709 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482401/remote/smithi033 2023-12-07T22:02:57.709 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-07T22:02:57.730 INFO:teuthology.task.internal:Removing archive directory... 2023-12-07T22:02:57.730 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-07T22:02:57.736 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-07T22:02:57.747 INFO:teuthology.task.internal:Not uploading archives. 2023-12-07T22:02:57.747 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-07T22:02:57.761 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-07T22:02:57.761 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-07T22:02:57.785 INFO:teuthology.orchestra.run.smithi033.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 7 22:02 /home/ubuntu/cephtest 2023-12-07T22:02:57.788 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-07T22:02:57.801 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-07T22:02:57.829 INFO:teuthology.nuke:Checking targets against current locks 2023-12-07T22:02:57.856 DEBUG:teuthology.nuke:shortname: smithi033 2023-12-07T22:02:57.857 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-07T22:02:57.883 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-07_21:36:14-rgw-wip-yuri5-testing-2023-12-07-0740-pacific-distro-default-smithi/7482401', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-07 21:38:41.080999', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-12-07T22:02:57.936 INFO:teuthology.orchestra.console.smithi033:Power off 2023-12-07T22:02:57.937 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-07T22:02:57.959 DEBUG:teuthology.orchestra.console.smithi033:power off output: Chassis Power Control: Down/Off 2023-12-07T22:02:57.960 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T22:02:57.982 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2023-12-07T22:03:01.984 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T22:03:02.095 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2023-12-07T22:03:06.098 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T22:03:06.208 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2023-12-07T22:03:10.210 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-07T22:03:10.319 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is off 2023-12-07T22:03:10.420 INFO:teuthology.orchestra.console.smithi033:Power off completed 2023-12-07T22:03:10.525 INFO:teuthology.run:Summary data: description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} duration: 944.5698883533478 failure_reason: 'Command failed (s3 tests against rgw) on smithi033 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a ''!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!s3select,!test_of_sts,!webidentity_test,!fails_with_subdomain''"' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7a7acadc3f1a4d69a5223f2f16d6e2bf status: fail success: false 2023-12-07T22:03:10.526 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-07T22:03:10.600 INFO:teuthology.run:FAIL