2023-01-24T16:48:35.521 INFO:root:teuthology version: 0.0.1.dev22+g8e1bba1 2023-01-24T16:48:35.522 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --description rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} --archive /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135469 --name yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi --verbose -- /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135469/orig.config.yaml 2023-01-24T16:48:35.555 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-24T16:48:35.611 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135469 branch: wip-yuri7-testing-2023-01-23-1532-quincy description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} email: yweinstein@radhat.com first_in_suite: false job_id: '7135469' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri7-testing-2023-01-23-1532-quincy ceph: conf: client: debug rgw: 20 rgw crypt kmip ca path: /etc/ceph/kmiproot.crt rgw crypt kmip client cert: /etc/ceph/kmip-client.crt rgw crypt kmip client key: /etc/ceph/kmip-client.key rgw crypt kmip kms key template: pykmip-$keyid rgw crypt require ssl: false rgw crypt s3 kms backend: kmip setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 49f8fb05584886826e8eade75f7105fba754560c wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 install: ceph: flavor: default sha1: 49f8fb05584886826e8eade75f7105fba754560c rgw: client.0: use-pykmip-role: client.0 workunit: branch: wip-yuri7-testing-2023-01-23-1532-quincy sha1: 49f8fb05584886826e8eade75f7105fba754560c owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 49f8fb05584886826e8eade75f7105fba754560c sleep_before_teardown: 0 suite: rgw suite_branch: wip-yuri7-testing-2023-01-23-1532-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 49f8fb05584886826e8eade75f7105fba754560c targets: smithi103.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= tasks: - install: null - ceph: null - openssl_keys: kmip-client: ca: kmiproot client: client.0 cn: rgw-client kmip-server: ca: kmiproot client: client.0 kmiproot: client: client.0 cn: kmiproot key-type: rsa:4096 - exec: client.0: - chmod 644 /home/ubuntu/cephtest/ca/kmip-client.key - pykmip: client.0: clientca: kmiproot clientcert: kmip-client secrets: - name: pykmip-my-key-1 - name: pykmip-my-key-2 servercert: kmip-server - rgw: client.0: null - s3tests: client.0: barbican: kms_key: my-key-1 kms_key2: my-key-2 force-branch: ceph-quincy vault_kv: key_path: my-key-1 key_path2: my-key-2 vault_old: key_path: my-key-1/1 key_path2: my-key-2/1 vault_transit: key_path: my-key-1 key_path2: my-key-2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8e1bba12e584b7ae912e92e94afe9c075ab51fa3 timestamp: 2023-01-24_16:04:03 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.17735 2023-01-24T16:48:35.612 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa; will attempt to use it 2023-01-24T16:48:35.612 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa/tasks 2023-01-24T16:48:35.613 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-24T16:48:35.615 INFO:teuthology.task.internal:Checking packages... 2023-01-24T16:48:35.641 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '49f8fb05584886826e8eade75f7105fba754560c' 2023-01-24T16:48:35.642 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-24T16:48:35.642 INFO:teuthology.packaging:ref: None 2023-01-24T16:48:35.642 INFO:teuthology.packaging:tag: None 2023-01-24T16:48:35.643 INFO:teuthology.packaging:branch: wip-yuri7-testing-2023-01-23-1532-quincy 2023-01-24T16:48:35.643 INFO:teuthology.packaging:sha1: 49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T16:48:35.643 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri7-testing-2023-01-23-1532-quincy 2023-01-24T16:48:35.797 INFO:teuthology.task.internal:Found packages for ceph version 17.2.5-727-g49f8fb05-1focal 2023-01-24T16:48:35.797 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-24T16:48:35.805 INFO:teuthology.task.internal:no buildpackages task found 2023-01-24T16:48:35.805 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-24T16:48:35.837 INFO:teuthology.task.internal:Saving configuration 2023-01-24T16:48:35.853 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-24T16:48:35.871 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-24T16:48:35.893 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135469', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-24 16:42:15.220208', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-01-24T16:48:35.893 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-24T16:48:35.904 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-01-24T16:48:35.904 INFO:teuthology.run_tasks:Running task console_log... 2023-01-24T16:48:35.967 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2f09ef65e0>, signals=[15]) 2023-01-24T16:48:35.968 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-24T16:48:35.975 INFO:teuthology.task.internal:Opening connections... 2023-01-24T16:48:35.975 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2023-01-24T16:48:35.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T16:48:36.024 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-24T16:48:36.031 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2023-01-24T16:48:36.146 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2023-01-24T16:48:36.147 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2023-01-24T16:48:36.196 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="Ubuntu" 2023-01-24T16:48:36.196 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-24T16:48:36.196 INFO:teuthology.orchestra.run.smithi103.stdout:ID=ubuntu 2023-01-24T16:48:36.197 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE=debian 2023-01-24T16:48:36.197 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-24T16:48:36.197 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="20.04" 2023-01-24T16:48:36.197 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-24T16:48:36.197 INFO:teuthology.orchestra.run.smithi103.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-24T16:48:36.198 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-24T16:48:36.198 INFO:teuthology.orchestra.run.smithi103.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-24T16:48:36.198 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_CODENAME=focal 2023-01-24T16:48:36.198 INFO:teuthology.orchestra.run.smithi103.stdout:UBUNTU_CODENAME=focal 2023-01-24T16:48:36.199 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2023-01-24T16:48:36.214 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-24T16:48:36.223 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-24T16:48:36.246 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-24T16:48:36.246 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2023-01-24T16:48:36.252 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-24T16:48:36.279 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-24T16:48:36.279 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2023-01-24T16:48:36.288 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-24T16:48:36.336 INFO:teuthology.run_tasks:Running task kernel... 2023-01-24T16:48:36.353 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-01-24T16:48:36.353 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-01-24T16:48:36.354 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-24T16:48:36.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T16:48:36.361 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2023-01-24T16:48:36.367 INFO:teuthology.orchestra.run.smithi103.stdout:5.4.0-124-generic 2023-01-24T16:48:36.367 INFO:teuthology.task.kernel:Running kernel on smithi103: 5.4.0-124-generic 2023-01-24T16:48:36.367 DEBUG:teuthology.orchestra.run.smithi103:> sudo apt-get clean 2023-01-24T16:48:36.467 DEBUG:teuthology.orchestra.run.smithi103:> sudo apt-get update 2023-01-24T16:48:36.627 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-24T16:48:36.640 INFO:teuthology.orchestra.run.smithi103.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-01-24T16:48:36.700 INFO:teuthology.orchestra.run.smithi103.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-01-24T16:48:36.733 INFO:teuthology.orchestra.run.smithi103.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-01-24T16:48:37.032 INFO:teuthology.orchestra.run.smithi103.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [775 kB] 2023-01-24T16:48:37.120 INFO:teuthology.orchestra.run.smithi103.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,336 kB] 2023-01-24T16:48:37.155 INFO:teuthology.orchestra.run.smithi103.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [403 kB] 2023-01-24T16:48:37.158 INFO:teuthology.orchestra.run.smithi103.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.2 kB] 2023-01-24T16:48:37.159 INFO:teuthology.orchestra.run.smithi103.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,560 kB] 2023-01-24T16:48:37.169 INFO:teuthology.orchestra.run.smithi103.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [30.4 kB] 2023-01-24T16:48:37.170 INFO:teuthology.orchestra.run.smithi103.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [220 kB] 2023-01-24T16:48:37.171 INFO:teuthology.orchestra.run.smithi103.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [620 B] 2023-01-24T16:48:37.171 INFO:teuthology.orchestra.run.smithi103.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [708 kB] 2023-01-24T16:48:37.176 INFO:teuthology.orchestra.run.smithi103.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,018 kB] 2023-01-24T16:48:37.183 INFO:teuthology.orchestra.run.smithi103.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [236 kB] 2023-01-24T16:48:37.185 INFO:teuthology.orchestra.run.smithi103.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [23.2 kB] 2023-01-24T16:48:37.189 INFO:teuthology.orchestra.run.smithi103.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.2 kB] 2023-01-24T16:48:37.190 INFO:teuthology.orchestra.run.smithi103.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,408 B] 2023-01-24T16:48:37.190 INFO:teuthology.orchestra.run.smithi103.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [604 B] 2023-01-24T16:48:37.190 INFO:teuthology.orchestra.run.smithi103.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-01-24T16:48:37.190 INFO:teuthology.orchestra.run.smithi103.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [24.9 kB] 2023-01-24T16:48:37.192 INFO:teuthology.orchestra.run.smithi103.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-01-24T16:48:37.350 INFO:teuthology.orchestra.run.smithi103.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [543 kB] 2023-01-24T16:48:37.645 INFO:teuthology.orchestra.run.smithi103.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,969 kB] 2023-01-24T16:48:37.823 INFO:teuthology.orchestra.run.smithi103.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [320 kB] 2023-01-24T16:48:37.836 INFO:teuthology.orchestra.run.smithi103.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [11.7 kB] 2023-01-24T16:48:37.837 INFO:teuthology.orchestra.run.smithi103.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [29.1 kB] 2023-01-24T16:48:37.838 INFO:teuthology.orchestra.run.smithi103.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,463 kB] 2023-01-24T16:48:37.896 INFO:teuthology.orchestra.run.smithi103.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [207 kB] 2023-01-24T16:48:37.901 INFO:teuthology.orchestra.run.smithi103.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [624 B] 2023-01-24T16:48:37.902 INFO:teuthology.orchestra.run.smithi103.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [576 kB] 2023-01-24T16:48:37.915 INFO:teuthology.orchestra.run.smithi103.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [787 kB] 2023-01-24T16:48:37.931 INFO:teuthology.orchestra.run.smithi103.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [152 kB] 2023-01-24T16:48:37.933 INFO:teuthology.orchestra.run.smithi103.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [16.9 kB] 2023-01-24T16:48:40.534 INFO:teuthology.orchestra.run.smithi103.stdout:Fetched 13.8 MB in 2s (6,509 kB/s) 2023-01-24T16:48:41.913 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2023-01-24T16:48:41.938 DEBUG:teuthology.orchestra.run.smithi103:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-01-24T16:48:42.009 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2023-01-24T16:48:42.195 INFO:teuthology.orchestra.run.smithi103.stdout:Building dependency tree... 2023-01-24T16:48:42.196 INFO:teuthology.orchestra.run.smithi103.stdout:Reading state information... 2023-01-24T16:48:42.357 INFO:teuthology.orchestra.run.smithi103.stdout:The following additional packages will be installed: 2023-01-24T16:48:42.357 INFO:teuthology.orchestra.run.smithi103.stdout: linux-generic linux-headers-5.4.0-137 linux-headers-5.4.0-137-generic 2023-01-24T16:48:42.357 INFO:teuthology.orchestra.run.smithi103.stdout: linux-headers-generic linux-image-5.4.0-137-generic 2023-01-24T16:48:42.357 INFO:teuthology.orchestra.run.smithi103.stdout: linux-modules-5.4.0-137-generic linux-modules-extra-5.4.0-137-generic 2023-01-24T16:48:42.358 INFO:teuthology.orchestra.run.smithi103.stdout:Suggested packages: 2023-01-24T16:48:42.358 INFO:teuthology.orchestra.run.smithi103.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-01-24T16:48:42.358 INFO:teuthology.orchestra.run.smithi103.stdout:Recommended packages: 2023-01-24T16:48:42.358 INFO:teuthology.orchestra.run.smithi103.stdout: thermald 2023-01-24T16:48:42.414 INFO:teuthology.orchestra.run.smithi103.stdout:The following NEW packages will be installed: 2023-01-24T16:48:42.415 INFO:teuthology.orchestra.run.smithi103.stdout: linux-headers-5.4.0-137 linux-headers-5.4.0-137-generic 2023-01-24T16:48:42.415 INFO:teuthology.orchestra.run.smithi103.stdout: linux-image-5.4.0-137-generic linux-modules-5.4.0-137-generic 2023-01-24T16:48:42.415 INFO:teuthology.orchestra.run.smithi103.stdout: linux-modules-extra-5.4.0-137-generic 2023-01-24T16:48:42.415 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages will be upgraded: 2023-01-24T16:48:42.416 INFO:teuthology.orchestra.run.smithi103.stdout: linux-generic linux-headers-generic linux-image-generic 2023-01-24T16:48:42.593 INFO:teuthology.orchestra.run.smithi103.stdout:3 upgraded, 5 newly installed, 0 to remove and 125 not upgraded. 2023-01-24T16:48:42.593 INFO:teuthology.orchestra.run.smithi103.stdout:Need to get 77.1 MB of archives. 2023-01-24T16:48:42.593 INFO:teuthology.orchestra.run.smithi103.stdout:After this operation, 380 MB of additional disk space will be used. 2023-01-24T16:48:42.593 INFO:teuthology.orchestra.run.smithi103.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-137-generic amd64 5.4.0-137.154 [15.0 MB] 2023-01-24T16:48:43.700 INFO:teuthology.orchestra.run.smithi103.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-137-generic amd64 5.4.0-137.154 [10.5 MB] 2023-01-24T16:48:43.966 INFO:teuthology.orchestra.run.smithi103.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-137-generic amd64 5.4.0-137.154 [39.2 MB] 2023-01-24T16:48:45.194 INFO:teuthology.orchestra.run.smithi103.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.137.135 [1,896 B] 2023-01-24T16:48:45.194 INFO:teuthology.orchestra.run.smithi103.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.137.135 [2,528 B] 2023-01-24T16:48:45.195 INFO:teuthology.orchestra.run.smithi103.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-137 all 5.4.0-137.154 [11.0 MB] 2023-01-24T16:48:45.464 INFO:teuthology.orchestra.run.smithi103.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-137-generic amd64 5.4.0-137.154 [1,400 kB] 2023-01-24T16:48:45.506 INFO:teuthology.orchestra.run.smithi103.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.137.135 [2,392 B] 2023-01-24T16:48:46.252 INFO:teuthology.orchestra.run.smithi103.stdout:Fetched 77.1 MB in 3s (24.8 MB/s) 2023-01-24T16:48:46.454 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package linux-modules-5.4.0-137-generic. 2023-01-24T16:48:48.211 INFO:teuthology.orchestra.run.smithi103.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-01-24T16:48:48.215 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../0-linux-modules-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-24T16:48:48.335 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking linux-modules-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T16:48:50.321 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package linux-image-5.4.0-137-generic. 2023-01-24T16:48:50.335 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../1-linux-image-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-24T16:48:50.440 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T16:48:51.524 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-137-generic. 2023-01-24T16:48:51.538 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-24T16:48:51.591 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking linux-modules-extra-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T16:48:56.560 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../3-linux-generic_5.4.0.137.135_amd64.deb ... 2023-01-24T16:48:56.685 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking linux-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-24T16:48:57.051 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.137.135_amd64.deb ... 2023-01-24T16:48:57.205 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking linux-image-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-24T16:48:57.567 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package linux-headers-5.4.0-137. 2023-01-24T16:48:57.581 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../5-linux-headers-5.4.0-137_5.4.0-137.154_all.deb ... 2023-01-24T16:48:57.625 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking linux-headers-5.4.0-137 (5.4.0-137.154) ... 2023-01-24T16:49:03.207 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package linux-headers-5.4.0-137-generic. 2023-01-24T16:49:03.229 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../6-linux-headers-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-24T16:49:03.273 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking linux-headers-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T16:49:07.187 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.137.135_amd64.deb ... 2023-01-24T16:49:07.343 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking linux-headers-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-24T16:49:07.701 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up linux-headers-5.4.0-137 (5.4.0-137.154) ... 2023-01-24T16:49:07.822 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up linux-modules-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T16:49:09.049 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up linux-headers-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T16:49:09.175 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T16:49:10.359 INFO:teuthology.orchestra.run.smithi103.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-137-generic 2023-01-24T16:49:10.359 INFO:teuthology.orchestra.run.smithi103.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-137-generic 2023-01-24T16:49:10.503 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up linux-modules-extra-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T16:49:12.288 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up linux-image-generic (5.4.0.137.135) ... 2023-01-24T16:49:12.439 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up linux-headers-generic (5.4.0.137.135) ... 2023-01-24T16:49:12.565 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up linux-generic (5.4.0.137.135) ... 2023-01-24T16:49:12.708 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-24T16:49:12.794 INFO:teuthology.orchestra.run.smithi103.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-01-24T16:49:12.794 INFO:teuthology.orchestra.run.smithi103.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-137-generic 2023-01-24T16:49:35.027 INFO:teuthology.orchestra.run.smithi103.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-01-24T16:49:35.028 INFO:teuthology.orchestra.run.smithi103.stdout:Sourcing file `/etc/default/grub' 2023-01-24T16:49:35.035 INFO:teuthology.orchestra.run.smithi103.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-01-24T16:49:35.051 INFO:teuthology.orchestra.run.smithi103.stdout:Generating grub configuration file ... 2023-01-24T16:49:35.391 INFO:teuthology.orchestra.run.smithi103.stdout:Found linux image: /boot/vmlinuz-5.4.0-137-generic 2023-01-24T16:49:35.414 INFO:teuthology.orchestra.run.smithi103.stdout:Found initrd image: /boot/initrd.img-5.4.0-137-generic 2023-01-24T16:49:35.784 INFO:teuthology.orchestra.run.smithi103.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-01-24T16:49:35.795 INFO:teuthology.orchestra.run.smithi103.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-01-24T16:49:38.559 INFO:teuthology.orchestra.run.smithi103.stdout:done 2023-01-24T16:49:39.020 DEBUG:teuthology.orchestra.run.smithi103:> dpkg -s linux-image-generic 2023-01-24T16:49:39.086 INFO:teuthology.orchestra.run.smithi103.stdout:Package: linux-image-generic 2023-01-24T16:49:39.087 INFO:teuthology.orchestra.run.smithi103.stdout:Status: install ok installed 2023-01-24T16:49:39.087 INFO:teuthology.orchestra.run.smithi103.stdout:Priority: optional 2023-01-24T16:49:39.087 INFO:teuthology.orchestra.run.smithi103.stdout:Section: kernel 2023-01-24T16:49:39.087 INFO:teuthology.orchestra.run.smithi103.stdout:Installed-Size: 19 2023-01-24T16:49:39.088 INFO:teuthology.orchestra.run.smithi103.stdout:Maintainer: Ubuntu Kernel Team 2023-01-24T16:49:39.088 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: amd64 2023-01-24T16:49:39.088 INFO:teuthology.orchestra.run.smithi103.stdout:Source: linux-meta 2023-01-24T16:49:39.088 INFO:teuthology.orchestra.run.smithi103.stdout:Version: 5.4.0.137.135 2023-01-24T16:49:39.088 INFO:teuthology.orchestra.run.smithi103.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2023-01-24T16:49:39.089 INFO:teuthology.orchestra.run.smithi103.stdout:Depends: linux-image-5.4.0-137-generic, linux-modules-extra-5.4.0-137-generic, linux-firmware, intel-microcode, amd64-microcode 2023-01-24T16:49:39.089 INFO:teuthology.orchestra.run.smithi103.stdout:Recommends: thermald 2023-01-24T16:49:39.089 INFO:teuthology.orchestra.run.smithi103.stdout:Description: Generic Linux kernel image 2023-01-24T16:49:39.089 INFO:teuthology.orchestra.run.smithi103.stdout: This package will always depend on the latest generic kernel image 2023-01-24T16:49:39.089 INFO:teuthology.orchestra.run.smithi103.stdout: available. 2023-01-24T16:49:39.090 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-137-generic 2023-01-24T16:49:39.090 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-24T16:49:39.090 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-24T16:49:39.090 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-01-24T16:49:39.091 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=None, version=distro) 2023-01-24T16:49:39.091 DEBUG:teuthology.orchestra.run.smithi103:> sudo apt-get clean 2023-01-24T16:49:39.161 DEBUG:teuthology.orchestra.run.smithi103:> sudo apt-get update 2023-01-24T16:49:39.294 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-01-24T16:49:39.411 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-24T16:49:39.491 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-01-24T16:49:39.571 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-01-24T16:49:40.935 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2023-01-24T16:49:40.958 DEBUG:teuthology.orchestra.run.smithi103:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-01-24T16:49:41.027 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2023-01-24T16:49:41.193 INFO:teuthology.orchestra.run.smithi103.stdout:Building dependency tree... 2023-01-24T16:49:41.193 INFO:teuthology.orchestra.run.smithi103.stdout:Reading state information... 2023-01-24T16:49:41.380 INFO:teuthology.orchestra.run.smithi103.stdout:linux-image-generic is already the newest version (5.4.0.137.135). 2023-01-24T16:49:41.380 INFO:teuthology.orchestra.run.smithi103.stdout:0 upgraded, 0 newly installed, 0 to remove and 125 not upgraded. 2023-01-24T16:49:41.382 DEBUG:teuthology.orchestra.run.smithi103:> dpkg -s linux-image-generic 2023-01-24T16:49:41.403 INFO:teuthology.orchestra.run.smithi103.stdout:Package: linux-image-generic 2023-01-24T16:49:41.403 INFO:teuthology.orchestra.run.smithi103.stdout:Status: install ok installed 2023-01-24T16:49:41.404 INFO:teuthology.orchestra.run.smithi103.stdout:Priority: optional 2023-01-24T16:49:41.404 INFO:teuthology.orchestra.run.smithi103.stdout:Section: kernel 2023-01-24T16:49:41.404 INFO:teuthology.orchestra.run.smithi103.stdout:Installed-Size: 19 2023-01-24T16:49:41.404 INFO:teuthology.orchestra.run.smithi103.stdout:Maintainer: Ubuntu Kernel Team 2023-01-24T16:49:41.404 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: amd64 2023-01-24T16:49:41.405 INFO:teuthology.orchestra.run.smithi103.stdout:Source: linux-meta 2023-01-24T16:49:41.405 INFO:teuthology.orchestra.run.smithi103.stdout:Version: 5.4.0.137.135 2023-01-24T16:49:41.405 INFO:teuthology.orchestra.run.smithi103.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2023-01-24T16:49:41.405 INFO:teuthology.orchestra.run.smithi103.stdout:Depends: linux-image-5.4.0-137-generic, linux-modules-extra-5.4.0-137-generic, linux-firmware, intel-microcode, amd64-microcode 2023-01-24T16:49:41.405 INFO:teuthology.orchestra.run.smithi103.stdout:Recommends: thermald 2023-01-24T16:49:41.406 INFO:teuthology.orchestra.run.smithi103.stdout:Description: Generic Linux kernel image 2023-01-24T16:49:41.406 INFO:teuthology.orchestra.run.smithi103.stdout: This package will always depend on the latest generic kernel image 2023-01-24T16:49:41.406 INFO:teuthology.orchestra.run.smithi103.stdout: available. 2023-01-24T16:49:41.406 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2023-01-24T16:49:41.453 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.4XCWZyVK1y 2023-01-24T16:49:41.453 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub/grub.cfg /tmp/tmp.4XCWZyVK1y 2023-01-24T16:49:41.466 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.4XCWZyVK1y 2023-01-24T16:49:41.552 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.4XCWZyVK1y is 9KB 2023-01-24T16:49:41.567 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.4XCWZyVK1y 2023-01-24T16:49:41.574 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-01-24T16:49:41.630 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:49:41.630 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-01-24T16:49:41.631 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-01-24T16:49:41.662 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-137-generic 2023-01-24T16:49:41.662 DEBUG:teuthology.orchestra.run.smithi103:> sudo update-grub 2023-01-24T16:49:42.959 INFO:teuthology.orchestra.run.smithi103.stderr:Sourcing file `/etc/default/grub' 2023-01-24T16:49:42.967 INFO:teuthology.orchestra.run.smithi103.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-01-24T16:49:42.977 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2023-01-24T16:49:43.242 INFO:teuthology.orchestra.run.smithi103.stderr:Found linux image: /boot/vmlinuz-5.4.0-137-generic 2023-01-24T16:49:43.254 INFO:teuthology.orchestra.run.smithi103.stderr:Found initrd image: /boot/initrd.img-5.4.0-137-generic 2023-01-24T16:49:43.639 INFO:teuthology.orchestra.run.smithi103.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-01-24T16:49:43.650 INFO:teuthology.orchestra.run.smithi103.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-01-24T16:49:45.267 INFO:teuthology.orchestra.run.smithi103.stderr:done 2023-01-24T16:49:45.268 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2023-01-24T16:50:15.300 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-01-24T16:50:15.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2023-01-24T16:50:15.301 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T16:50:33.708 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2023-01-24T16:51:03.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2023-01-24T16:51:03.737 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T16:51:35.503 DEBUG:teuthology.orchestra.remote:No existing session 2023-01-24T16:52:05.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2023-01-24T16:52:05.533 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T16:52:05.582 DEBUG:teuthology.orchestra.run.smithi103:> true 2023-01-24T16:52:06.207 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2023-01-24T16:52:06.207 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-137-generic"... 2023-01-24T16:52:06.207 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2023-01-24T16:52:06.214 INFO:teuthology.orchestra.run.smithi103.stdout:5.4.0-137-generic 2023-01-24T16:52:06.214 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-137-generic vs 5.4.0-137-generic 2023-01-24T16:52:06.214 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-24T16:52:06.215 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-01-24T16:52:07.216 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-01-24T16:52:07.217 DEBUG:teuthology.orchestra.run.smithi103:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-24T16:52:07.302 INFO:teuthology.orchestra.run.smithi103.stdout:ttyS1 2023-01-24T16:52:07.321 DEBUG:teuthology.parallel:result is None 2023-01-24T16:52:07.322 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-24T16:52:07.328 INFO:teuthology.task.internal:Creating test directory... 2023-01-24T16:52:07.329 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-24T16:52:07.336 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-24T16:52:07.373 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-24T16:52:07.429 INFO:teuthology.task.internal:Creating archive directory... 2023-01-24T16:52:07.429 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-24T16:52:07.446 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-24T16:52:07.474 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-24T16:52:07.475 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-24T16:52:07.494 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T16:52:07.504 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-24T16:52:07.506 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-24T16:52:07.524 INFO:teuthology.task.internal:Configuring sudo... 2023-01-24T16:52:07.525 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-24T16:52:07.561 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-24T16:52:07.576 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-24T16:52:07.576 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-24T16:52:07.583 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-24T16:52:07.589 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-24T16:52:07.637 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:52:07.638 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-24T16:52:07.659 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2023-01-24T16:52:07.701 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-24T16:52:07.709 INFO:teuthology.task.internal:Starting timer... 2023-01-24T16:52:07.709 INFO:teuthology.run_tasks:Running task pcp... 2023-01-24T16:52:07.744 INFO:teuthology.run_tasks:Running task selinux... 2023-01-24T16:52:07.802 DEBUG:teuthology.task.selinux:Excluding smithi103: OS 'ubuntu' does not support SELinux 2023-01-24T16:52:07.802 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-24T16:52:07.802 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-01-24T16:52:07.803 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-24T16:52:07.803 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-24T16:52:07.852 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-01-24T16:52:07.853 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-24T16:52:07.872 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-24T16:52:07.873 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi103.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-24T16:56:47.877 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi103.front.sepia.ceph.com')] 2023-01-24T16:56:47.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2023-01-24T16:56:47.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-24T16:56:47.933 DEBUG:teuthology.orchestra.run.smithi103:> true 2023-01-24T16:56:48.050 INFO:teuthology.run_tasks:Running task clock... 2023-01-24T16:56:48.061 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-01-24T16:56:48.061 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-24T16:56:48.062 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-24T16:56:48.094 INFO:teuthology.orchestra.run.smithi103.stdout:24 Jan 16:56:48 ntpd[19469]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-01-24T16:56:48.094 INFO:teuthology.orchestra.run.smithi103.stdout:24 Jan 16:56:48 ntpd[19469]: Command line: ntpd -gq 2023-01-24T16:56:48.095 INFO:teuthology.orchestra.run.smithi103.stdout:24 Jan 16:56:48 ntpd[19469]: proto: precision = 0.144 usec (-23) 2023-01-24T16:56:48.095 INFO:teuthology.orchestra.run.smithi103.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-01-24T16:56:48.096 INFO:teuthology.orchestra.run.smithi103.stdout:24 Jan 16:56:48 ntpd[19469]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-01-24T16:56:48.096 INFO:teuthology.orchestra.run.smithi103.stderr:restrict ::: KOD does nothing without LIMITED. 2023-01-24T16:56:48.097 INFO:teuthology.orchestra.run.smithi103.stdout:24 Jan 16:56:48 ntpd[19469]: restrict ::: KOD does nothing without LIMITED. 2023-01-24T16:56:48.097 INFO:teuthology.orchestra.run.smithi103.stdout:24 Jan 16:56:48 ntpd[19469]: Listen and drop on 0 v6wildcard [::]:123 2023-01-24T16:56:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:24 Jan 16:56:48 ntpd[19469]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-01-24T16:56:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:24 Jan 16:56:48 ntpd[19469]: Listen normally on 2 lo 127.0.0.1:123 2023-01-24T16:56:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:24 Jan 16:56:48 ntpd[19469]: Listen normally on 3 enp3s0f1 172.21.15.103:123 2023-01-24T16:56:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:24 Jan 16:56:48 ntpd[19469]: Listen normally on 4 lo [::1]:123 2023-01-24T16:56:48.098 INFO:teuthology.orchestra.run.smithi103.stdout:24 Jan 16:56:48 ntpd[19469]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cc0b%5]:123 2023-01-24T16:56:48.099 INFO:teuthology.orchestra.run.smithi103.stdout:24 Jan 16:56:48 ntpd[19469]: Listening on routing socket on fd #22 for interface updates 2023-01-24T16:56:49.094 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:49 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:49.095 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:49 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:49.095 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:49 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:49.095 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:49 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:49.095 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:49 ntpd[19469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T16:56:49.096 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:49 ntpd[19469]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T16:56:51.095 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:51 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:51.096 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:51 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:51.096 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:51 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:51.096 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:51 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:51.096 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:51 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:51.096 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:51 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:51.097 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:51 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:51.097 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:51 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:51.097 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:51 ntpd[19469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T16:56:51.097 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:51 ntpd[19469]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T16:56:51.098 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:51 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:51.098 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:51 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:51.098 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:51 ntpd[19469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T16:56:51.098 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:51 ntpd[19469]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T16:56:53.096 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:53.096 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:53.096 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:53.096 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:53.097 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:53.097 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:53.097 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:53.097 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:53.097 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T16:56:53.098 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T16:56:53.098 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:53.098 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:53.098 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T16:56:53.098 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T16:56:53.098 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:53.099 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:53.099 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T16:56:53.099 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:53 ntpd[19469]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T16:56:55.096 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:55.096 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:55.096 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:55.096 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:55.097 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:55.097 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:55.097 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:55.097 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:55.098 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T16:56:55.098 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T16:56:55.098 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:55.098 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:55.098 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T16:56:55.099 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T16:56:55.099 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-24T16:56:55.099 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: can't open /var/log/ntpstats/rawstats.20230124: Permission denied 2023-01-24T16:56:55.099 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-24T16:56:55.099 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: can't open /var/log/ntpstats/peerstats.20230124: Permission denied 2023-01-24T16:56:55.100 INFO:teuthology.orchestra.run.smithi103.stdout:24 Jan 16:56:55 ntpd[19469]: ntpd: time slew -0.004630 s 2023-01-24T16:56:55.100 INFO:teuthology.orchestra.run.smithi103.stdout:ntpd: time slew -0.004630s 2023-01-24T16:56:55.101 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-01-24T16:56:55.101 INFO:teuthology.orchestra.run.smithi103.stderr:24 Jan 16:56:55 ntpd[19469]: can't open /var/log/ntpstats/loopstats.20230124: Permission denied 2023-01-24T16:56:55.169 INFO:teuthology.orchestra.run.smithi103.stdout: remote refid st t when poll reach delay offset jitter 2023-01-24T16:56:55.169 INFO:teuthology.orchestra.run.smithi103.stdout:============================================================================== 2023-01-24T16:56:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-24T16:56:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-24T16:56:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-24T16:56:55.170 INFO:teuthology.orchestra.run.smithi103.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-24T16:56:55.171 INFO:teuthology.run_tasks:Running task install... 2023-01-24T16:56:55.183 DEBUG:teuthology.task.install:project ceph 2023-01-24T16:56:55.184 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'}} 2023-01-24T16:56:55.184 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'} 2023-01-24T16:56:55.184 INFO:teuthology.task.install:Using flavor: default 2023-01-24T16:56:55.192 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-01-24T16:56:55.192 INFO:teuthology.task.install:extra packages: [] 2023-01-24T16:56:55.193 DEBUG:teuthology.orchestra.run.smithi103:> sudo lsb_release -is 2023-01-24T16:56:55.260 INFO:teuthology.orchestra.run.smithi103.stdout:Ubuntu 2023-01-24T16:56:55.266 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-24T16:56:55.266 DEBUG:teuthology.orchestra.run.smithi103:> sudo apt-key list | grep Ceph 2023-01-24T16:56:55.364 INFO:teuthology.orchestra.run.smithi103.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-01-24T16:56:55.421 INFO:teuthology.orchestra.run.smithi103.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-01-24T16:56:55.421 INFO:teuthology.orchestra.run.smithi103.stdout:uid [ unknown] Ceph.com (release key) 2023-01-24T16:56:55.423 DEBUG:teuthology.orchestra.run.smithi103:> sudo lsb_release -is 2023-01-24T16:56:55.471 INFO:teuthology.orchestra.run.smithi103.stdout:Ubuntu 2023-01-24T16:56:55.476 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-24T16:56:55.477 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-01-24T16:56:55.477 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T16:56:55.643 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default/ 2023-01-24T16:56:55.643 INFO:teuthology.task.install.deb:Package version is 17.2.5-727-g49f8fb05-1focal 2023-01-24T16:56:55.789 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:56:55.790 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-01-24T16:56:55.806 DEBUG:teuthology.orchestra.run.smithi103:> sudo apt-get update 2023-01-24T16:56:56.051 INFO:teuthology.orchestra.run.smithi103.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal InRelease 2023-01-24T16:56:56.075 INFO:teuthology.orchestra.run.smithi103.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-01-24T16:56:56.100 INFO:teuthology.orchestra.run.smithi103.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal Release.gpg 2023-01-24T16:56:56.116 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-24T16:56:56.116 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-01-24T16:56:56.124 INFO:teuthology.orchestra.run.smithi103.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-01-24T16:56:56.160 INFO:teuthology.orchestra.run.smithi103.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main i386 Packages [3,372 B] 2023-01-24T16:56:56.184 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-01-24T16:56:56.264 INFO:teuthology.orchestra.run.smithi103.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-01-24T16:56:56.768 INFO:teuthology.orchestra.run.smithi103.stdout:Fetched 28.4 kB in 1s (37.8 kB/s) 2023-01-24T16:56:57.709 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2023-01-24T16:56:57.733 DEBUG:teuthology.orchestra.run.smithi103:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.5-727-g49f8fb05-1focal cephadm=17.2.5-727-g49f8fb05-1focal ceph-mds=17.2.5-727-g49f8fb05-1focal ceph-mgr=17.2.5-727-g49f8fb05-1focal ceph-common=17.2.5-727-g49f8fb05-1focal ceph-fuse=17.2.5-727-g49f8fb05-1focal ceph-test=17.2.5-727-g49f8fb05-1focal ceph-volume=17.2.5-727-g49f8fb05-1focal radosgw=17.2.5-727-g49f8fb05-1focal python3-rados=17.2.5-727-g49f8fb05-1focal python3-rgw=17.2.5-727-g49f8fb05-1focal python3-cephfs=17.2.5-727-g49f8fb05-1focal python3-rbd=17.2.5-727-g49f8fb05-1focal libcephfs2=17.2.5-727-g49f8fb05-1focal libcephfs-dev=17.2.5-727-g49f8fb05-1focal librados2=17.2.5-727-g49f8fb05-1focal librbd1=17.2.5-727-g49f8fb05-1focal rbd-fuse=17.2.5-727-g49f8fb05-1focal 2023-01-24T16:56:57.801 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2023-01-24T16:56:57.986 INFO:teuthology.orchestra.run.smithi103.stdout:Building dependency tree... 2023-01-24T16:56:57.986 INFO:teuthology.orchestra.run.smithi103.stdout:Reading state information... 2023-01-24T16:56:58.143 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages were automatically installed and are no longer required: 2023-01-24T16:56:58.144 INFO:teuthology.orchestra.run.smithi103.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-01-24T16:56:58.144 INFO:teuthology.orchestra.run.smithi103.stdout:Use 'sudo apt autoremove' to remove them. 2023-01-24T16:56:58.144 INFO:teuthology.orchestra.run.smithi103.stdout:The following additional packages will be installed: 2023-01-24T16:56:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-01-24T16:56:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-01-24T16:56:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-01-24T16:56:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-01-24T16:56:58.145 INFO:teuthology.orchestra.run.smithi103.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-01-24T16:56:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-01-24T16:56:58.146 INFO:teuthology.orchestra.run.smithi103.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2023-01-24T16:56:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2023-01-24T16:56:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2023-01-24T16:56:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2023-01-24T16:56:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2023-01-24T16:56:58.147 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2023-01-24T16:56:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2023-01-24T16:56:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-01-24T16:56:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2023-01-24T16:56:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2023-01-24T16:56:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2023-01-24T16:56:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-01-24T16:56:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2023-01-24T16:56:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2023-01-24T16:56:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2023-01-24T16:56:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: socat xmlstarlet 2023-01-24T16:56:58.150 INFO:teuthology.orchestra.run.smithi103.stdout:Suggested packages: 2023-01-24T16:56:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-01-24T16:56:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-01-24T16:56:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-01-24T16:56:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-01-24T16:56:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-01-24T16:56:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-01-24T16:56:58.151 INFO:teuthology.orchestra.run.smithi103.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-01-24T16:56:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-01-24T16:56:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-01-24T16:56:58.152 INFO:teuthology.orchestra.run.smithi103.stdout:Recommended packages: 2023-01-24T16:56:58.152 INFO:teuthology.orchestra.run.smithi103.stdout: btrfs-tools 2023-01-24T16:56:58.259 INFO:teuthology.orchestra.run.smithi103.stdout:The following NEW packages will be installed: 2023-01-24T16:56:58.260 INFO:teuthology.orchestra.run.smithi103.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-01-24T16:56:58.260 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-01-24T16:56:58.260 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-01-24T16:56:58.260 INFO:teuthology.orchestra.run.smithi103.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-01-24T16:56:58.260 INFO:teuthology.orchestra.run.smithi103.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-01-24T16:56:58.261 INFO:teuthology.orchestra.run.smithi103.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-01-24T16:56:58.261 INFO:teuthology.orchestra.run.smithi103.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-01-24T16:56:58.262 INFO:teuthology.orchestra.run.smithi103.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2023-01-24T16:56:58.262 INFO:teuthology.orchestra.run.smithi103.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2023-01-24T16:56:58.262 INFO:teuthology.orchestra.run.smithi103.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-01-24T16:56:58.263 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2023-01-24T16:56:58.263 INFO:teuthology.orchestra.run.smithi103.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2023-01-24T16:56:58.263 INFO:teuthology.orchestra.run.smithi103.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2023-01-24T16:56:58.263 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2023-01-24T16:56:58.263 INFO:teuthology.orchestra.run.smithi103.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2023-01-24T16:56:58.264 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2023-01-24T16:56:58.264 INFO:teuthology.orchestra.run.smithi103.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-01-24T16:56:58.264 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2023-01-24T16:56:58.264 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2023-01-24T16:56:58.264 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2023-01-24T16:56:58.265 INFO:teuthology.orchestra.run.smithi103.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-01-24T16:56:58.265 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-01-24T16:56:58.265 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-01-24T16:56:58.265 INFO:teuthology.orchestra.run.smithi103.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2023-01-24T16:56:58.265 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages will be upgraded: 2023-01-24T16:56:58.266 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 librbd1 2023-01-24T16:56:58.418 INFO:teuthology.orchestra.run.smithi103.stdout:2 upgraded, 108 newly installed, 0 to remove and 120 not upgraded. 2023-01-24T16:56:58.418 INFO:teuthology.orchestra.run.smithi103.stdout:Need to get 209 MB of archives. 2023-01-24T16:56:58.418 INFO:teuthology.orchestra.run.smithi103.stdout:After this operation, 980 MB of additional disk space will be used. 2023-01-24T16:56:58.419 INFO:teuthology.orchestra.run.smithi103.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.5-727-g49f8fb05-1focal [3,502 kB] 2023-01-24T16:56:58.450 INFO:teuthology.orchestra.run.smithi103.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-01-24T16:56:58.839 INFO:teuthology.orchestra.run.smithi103.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-01-24T16:56:58.846 INFO:teuthology.orchestra.run.smithi103.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-01-24T16:56:58.853 INFO:teuthology.orchestra.run.smithi103.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.5-727-g49f8fb05-1focal [3,550 kB] 2023-01-24T16:56:58.890 INFO:teuthology.orchestra.run.smithi103.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-01-24T16:56:58.903 INFO:teuthology.orchestra.run.smithi103.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-01-24T16:56:59.026 INFO:teuthology.orchestra.run.smithi103.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-01-24T16:56:59.032 INFO:teuthology.orchestra.run.smithi103.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-01-24T16:56:59.060 INFO:teuthology.orchestra.run.smithi103.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-01-24T16:56:59.064 INFO:teuthology.orchestra.run.smithi103.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.5-727-g49f8fb05-1focal [755 kB] 2023-01-24T16:56:59.104 INFO:teuthology.orchestra.run.smithi103.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.5-727-g49f8fb05-1focal [412 kB] 2023-01-24T16:56:59.124 INFO:teuthology.orchestra.run.smithi103.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.5-727-g49f8fb05-1focal [22.5 kB] 2023-01-24T16:56:59.125 INFO:teuthology.orchestra.run.smithi103.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.5-727-g49f8fb05-1focal [200 kB] 2023-01-24T16:56:59.129 INFO:teuthology.orchestra.run.smithi103.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.5-727-g49f8fb05-1focal [43.4 kB] 2023-01-24T16:56:59.130 INFO:teuthology.orchestra.run.smithi103.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.5-727-g49f8fb05-1focal [384 kB] 2023-01-24T16:56:59.149 INFO:teuthology.orchestra.run.smithi103.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.5-727-g49f8fb05-1focal [5,528 kB] 2023-01-24T16:56:59.199 INFO:teuthology.orchestra.run.smithi103.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-01-24T16:56:59.207 INFO:teuthology.orchestra.run.smithi103.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-01-24T16:56:59.230 INFO:teuthology.orchestra.run.smithi103.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-01-24T16:56:59.237 INFO:teuthology.orchestra.run.smithi103.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-01-24T16:56:59.238 INFO:teuthology.orchestra.run.smithi103.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-01-24T16:56:59.243 INFO:teuthology.orchestra.run.smithi103.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-01-24T16:56:59.244 INFO:teuthology.orchestra.run.smithi103.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-01-24T16:56:59.245 INFO:teuthology.orchestra.run.smithi103.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-01-24T16:56:59.249 INFO:teuthology.orchestra.run.smithi103.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-01-24T16:56:59.280 INFO:teuthology.orchestra.run.smithi103.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-01-24T16:56:59.331 INFO:teuthology.orchestra.run.smithi103.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-01-24T16:56:59.331 INFO:teuthology.orchestra.run.smithi103.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-01-24T16:56:59.332 INFO:teuthology.orchestra.run.smithi103.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-01-24T16:56:59.333 INFO:teuthology.orchestra.run.smithi103.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-01-24T16:56:59.334 INFO:teuthology.orchestra.run.smithi103.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-01-24T16:56:59.335 INFO:teuthology.orchestra.run.smithi103.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-01-24T16:56:59.336 INFO:teuthology.orchestra.run.smithi103.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-01-24T16:56:59.337 INFO:teuthology.orchestra.run.smithi103.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-01-24T16:56:59.339 INFO:teuthology.orchestra.run.smithi103.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.5-727-g49f8fb05-1focal [116 kB] 2023-01-24T16:56:59.341 INFO:teuthology.orchestra.run.smithi103.stdout:Get:37 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.5-727-g49f8fb05-1focal [479 kB] 2023-01-24T16:56:59.358 INFO:teuthology.orchestra.run.smithi103.stdout:Get:38 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.5-727-g49f8fb05-1focal [27.1 MB] 2023-01-24T16:56:59.359 INFO:teuthology.orchestra.run.smithi103.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-01-24T16:56:59.420 INFO:teuthology.orchestra.run.smithi103.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-01-24T16:56:59.421 INFO:teuthology.orchestra.run.smithi103.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-01-24T16:56:59.428 INFO:teuthology.orchestra.run.smithi103.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-01-24T16:56:59.429 INFO:teuthology.orchestra.run.smithi103.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-01-24T16:56:59.430 INFO:teuthology.orchestra.run.smithi103.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-01-24T16:56:59.430 INFO:teuthology.orchestra.run.smithi103.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-01-24T16:56:59.432 INFO:teuthology.orchestra.run.smithi103.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2023-01-24T16:56:59.435 INFO:teuthology.orchestra.run.smithi103.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-01-24T16:56:59.439 INFO:teuthology.orchestra.run.smithi103.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-01-24T16:56:59.516 INFO:teuthology.orchestra.run.smithi103.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-01-24T16:56:59.519 INFO:teuthology.orchestra.run.smithi103.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-01-24T16:56:59.521 INFO:teuthology.orchestra.run.smithi103.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-01-24T16:56:59.523 INFO:teuthology.orchestra.run.smithi103.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-01-24T16:56:59.524 INFO:teuthology.orchestra.run.smithi103.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-01-24T16:56:59.525 INFO:teuthology.orchestra.run.smithi103.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-01-24T16:56:59.791 INFO:teuthology.orchestra.run.smithi103.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-01-24T16:56:59.868 INFO:teuthology.orchestra.run.smithi103.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-01-24T16:56:59.978 INFO:teuthology.orchestra.run.smithi103.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-01-24T16:56:59.997 INFO:teuthology.orchestra.run.smithi103.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-01-24T16:56:59.998 INFO:teuthology.orchestra.run.smithi103.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-01-24T16:56:59.998 INFO:teuthology.orchestra.run.smithi103.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-01-24T16:56:59.999 INFO:teuthology.orchestra.run.smithi103.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-01-24T16:56:59.999 INFO:teuthology.orchestra.run.smithi103.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-01-24T16:56:59.999 INFO:teuthology.orchestra.run.smithi103.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-01-24T16:57:00.000 INFO:teuthology.orchestra.run.smithi103.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-01-24T16:57:00.000 INFO:teuthology.orchestra.run.smithi103.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-01-24T16:57:00.059 INFO:teuthology.orchestra.run.smithi103.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-01-24T16:57:00.081 INFO:teuthology.orchestra.run.smithi103.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-01-24T16:57:00.086 INFO:teuthology.orchestra.run.smithi103.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-01-24T16:57:00.089 INFO:teuthology.orchestra.run.smithi103.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-01-24T16:57:00.092 INFO:teuthology.orchestra.run.smithi103.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-01-24T16:57:00.094 INFO:teuthology.orchestra.run.smithi103.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-01-24T16:57:00.097 INFO:teuthology.orchestra.run.smithi103.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-01-24T16:57:00.099 INFO:teuthology.orchestra.run.smithi103.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-01-24T16:57:00.166 INFO:teuthology.orchestra.run.smithi103.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-01-24T16:57:00.170 INFO:teuthology.orchestra.run.smithi103.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-01-24T16:57:00.351 INFO:teuthology.orchestra.run.smithi103.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-01-24T16:57:00.426 INFO:teuthology.orchestra.run.smithi103.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-01-24T16:57:00.427 INFO:teuthology.orchestra.run.smithi103.stdout:Get:78 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.5-727-g49f8fb05-1focal [6,439 kB] 2023-01-24T16:57:00.427 INFO:teuthology.orchestra.run.smithi103.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-01-24T16:57:00.428 INFO:teuthology.orchestra.run.smithi103.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-01-24T16:57:00.428 INFO:teuthology.orchestra.run.smithi103.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-01-24T16:57:00.430 INFO:teuthology.orchestra.run.smithi103.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-01-24T16:57:00.448 INFO:teuthology.orchestra.run.smithi103.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-01-24T16:57:00.448 INFO:teuthology.orchestra.run.smithi103.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-01-24T16:57:00.449 INFO:teuthology.orchestra.run.smithi103.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-01-24T16:57:00.449 INFO:teuthology.orchestra.run.smithi103.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-01-24T16:57:00.512 INFO:teuthology.orchestra.run.smithi103.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-01-24T16:57:00.513 INFO:teuthology.orchestra.run.smithi103.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-01-24T16:57:00.514 INFO:teuthology.orchestra.run.smithi103.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-01-24T16:57:00.514 INFO:teuthology.orchestra.run.smithi103.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-01-24T16:57:00.517 INFO:teuthology.orchestra.run.smithi103.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-01-24T16:57:00.520 INFO:teuthology.orchestra.run.smithi103.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.5-727-g49f8fb05-1focal [218 kB] 2023-01-24T16:57:00.527 INFO:teuthology.orchestra.run.smithi103.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.5-727-g49f8fb05-1focal [152 kB] 2023-01-24T16:57:00.532 INFO:teuthology.orchestra.run.smithi103.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.5-727-g49f8fb05-1focal [1,583 kB] 2023-01-24T16:57:00.592 INFO:teuthology.orchestra.run.smithi103.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-01-24T16:57:00.666 INFO:teuthology.orchestra.run.smithi103.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.5-727-g49f8fb05-1focal [7,828 kB] 2023-01-24T16:57:00.848 INFO:teuthology.orchestra.run.smithi103.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.5-727-g49f8fb05-1focal [28.6 MB] 2023-01-24T16:57:01.886 INFO:teuthology.orchestra.run.smithi103.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.5-727-g49f8fb05-1focal [3,904 B] 2023-01-24T16:57:01.957 INFO:teuthology.orchestra.run.smithi103.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.5-727-g49f8fb05-1focal [865 kB] 2023-01-24T16:57:01.973 INFO:teuthology.orchestra.run.smithi103.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.5-727-g49f8fb05-1focal [2,387 kB] 2023-01-24T16:57:01.990 INFO:teuthology.orchestra.run.smithi103.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.5-727-g49f8fb05-1focal [77.0 kB] 2023-01-24T16:57:01.990 INFO:teuthology.orchestra.run.smithi103.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.5-727-g49f8fb05-1focal [111 kB] 2023-01-24T16:57:01.991 INFO:teuthology.orchestra.run.smithi103.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.5-727-g49f8fb05-1focal [1,713 kB] 2023-01-24T16:57:02.042 INFO:teuthology.orchestra.run.smithi103.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.5-727-g49f8fb05-1focal [7,878 kB] 2023-01-24T16:57:02.315 INFO:teuthology.orchestra.run.smithi103.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.5-727-g49f8fb05-1focal [4,020 B] 2023-01-24T16:57:02.316 INFO:teuthology.orchestra.run.smithi103.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.5-727-g49f8fb05-1focal [56.6 MB] 2023-01-24T16:57:04.387 INFO:teuthology.orchestra.run.smithi103.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.5-727-g49f8fb05-1focal [225 kB] 2023-01-24T16:57:04.391 INFO:teuthology.orchestra.run.smithi103.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.5-727-g49f8fb05-1focal [21.8 kB] 2023-01-24T16:57:04.392 INFO:teuthology.orchestra.run.smithi103.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.5-727-g49f8fb05-1focal [16.9 MB] 2023-01-24T16:57:04.912 INFO:teuthology.orchestra.run.smithi103.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2023-01-23-1532-quincy/49f8fb05584886826e8eade75f7105fba754560c/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.5-727-g49f8fb05-1focal [80.1 kB] 2023-01-24T16:57:05.527 INFO:teuthology.orchestra.run.smithi103.stdout:Fetched 209 MB in 7s (31.4 MB/s) 2023-01-24T16:57:05.691 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-01-24T16:57:05.747 INFO:teuthology.orchestra.run.smithi103.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141754 files and directories currently installed.) 2023-01-24T16:57:05.752 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-01-24T16:57:05.840 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-01-24T16:57:06.184 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../001-librbd1_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:06.319 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking librbd1 (17.2.5-727-g49f8fb05-1focal) over (15.2.17-0ubuntu0.20.04.1) ... 2023-01-24T16:57:06.964 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../002-librados2_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:07.065 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking librados2 (17.2.5-727-g49f8fb05-1focal) over (15.2.17-0ubuntu0.20.04.1) ... 2023-01-24T16:57:07.761 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libcephfs2. 2023-01-24T16:57:07.777 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../003-libcephfs2_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:07.819 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libcephfs2 (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:08.146 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-rados. 2023-01-24T16:57:08.163 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../004-python3-rados_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:08.205 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-rados (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:08.491 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-01-24T16:57:08.510 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.5-727-g49f8fb05-1focal_all.deb ... 2023-01-24T16:57:08.549 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-ceph-argparse (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:08.784 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-cephfs. 2023-01-24T16:57:08.803 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../006-python3-cephfs_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:08.843 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-cephfs (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:09.221 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-ceph-common. 2023-01-24T16:57:09.239 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.5-727-g49f8fb05-1focal_all.deb ... 2023-01-24T16:57:09.279 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-ceph-common (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:09.565 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-prettytable. 2023-01-24T16:57:09.584 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2023-01-24T16:57:09.623 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-01-24T16:57:09.868 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-rbd. 2023-01-24T16:57:09.884 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../009-python3-rbd_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:09.918 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-rbd (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:10.271 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-01-24T16:57:10.288 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-01-24T16:57:10.329 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-01-24T16:57:10.699 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-01-24T16:57:10.715 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2023-01-24T16:57:10.757 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-01-24T16:57:11.018 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-01-24T16:57:11.033 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-01-24T16:57:11.068 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-01-24T16:57:11.454 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-01-24T16:57:11.472 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-01-24T16:57:11.513 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-01-24T16:57:11.857 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-01-24T16:57:11.873 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-01-24T16:57:11.915 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-01-24T16:57:12.327 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-01-24T16:57:12.343 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-24T16:57:12.388 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-24T16:57:12.836 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-01-24T16:57:12.855 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-24T16:57:12.886 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-24T16:57:13.273 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-01-24T16:57:13.288 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-24T16:57:13.331 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-24T16:57:13.726 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-01-24T16:57:13.743 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-01-24T16:57:13.809 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-01-24T16:57:14.112 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package librgw2. 2023-01-24T16:57:14.131 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../019-librgw2_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:14.162 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking librgw2 (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:14.939 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-rgw. 2023-01-24T16:57:14.955 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../020-python3-rgw_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:14.998 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-rgw (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:15.351 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package liboath0. 2023-01-24T16:57:15.367 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-01-24T16:57:15.418 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-01-24T16:57:15.846 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libradosstriper1. 2023-01-24T16:57:15.863 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../022-libradosstriper1_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:15.904 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libradosstriper1 (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:16.391 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-common. 2023-01-24T16:57:16.407 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../023-ceph-common_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:16.449 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-common (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:19.163 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-base. 2023-01-24T16:57:19.179 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../024-ceph-base_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:19.232 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-base (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:20.217 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-cryptography. 2023-01-24T16:57:20.233 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-01-24T16:57:20.300 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-01-24T16:57:20.704 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-openssl. 2023-01-24T16:57:20.723 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2023-01-24T16:57:20.762 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-01-24T16:57:21.082 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-01-24T16:57:21.102 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_17.2.5-727-g49f8fb05-1focal_all.deb ... 2023-01-24T16:57:21.148 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-mgr-modules-core (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:21.495 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-bcrypt. 2023-01-24T16:57:21.514 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../028-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-01-24T16:57:21.545 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-01-24T16:57:21.831 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-cherrypy3. 2023-01-24T16:57:21.850 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../029-python3-cherrypy3_8.9.1-6_all.deb ... 2023-01-24T16:57:21.914 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-01-24T16:57:22.243 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-natsort. 2023-01-24T16:57:22.260 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../030-python3-natsort_7.0.1-1_all.deb ... 2023-01-24T16:57:22.301 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-01-24T16:57:22.621 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-jwt. 2023-01-24T16:57:22.639 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../031-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-01-24T16:57:22.704 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-01-24T16:57:22.999 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-logutils. 2023-01-24T16:57:23.015 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-7_all.deb ... 2023-01-24T16:57:23.057 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-01-24T16:57:23.346 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-markupsafe. 2023-01-24T16:57:23.365 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../033-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-01-24T16:57:23.404 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-01-24T16:57:23.741 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-mako. 2023-01-24T16:57:23.757 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../034-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-01-24T16:57:23.799 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-01-24T16:57:24.119 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-simplegeneric. 2023-01-24T16:57:24.136 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-3_all.deb ... 2023-01-24T16:57:24.177 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-01-24T16:57:24.497 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-singledispatch. 2023-01-24T16:57:24.512 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-01-24T16:57:24.546 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-01-24T16:57:24.849 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-webob. 2023-01-24T16:57:24.865 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../037-python3-webob_1%3a1.8.5-2_all.deb ... 2023-01-24T16:57:24.907 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-01-24T16:57:25.261 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-soupsieve. 2023-01-24T16:57:25.277 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../038-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-01-24T16:57:25.319 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-01-24T16:57:25.622 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-bs4. 2023-01-24T16:57:25.638 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../039-python3-bs4_4.8.2-1_all.deb ... 2023-01-24T16:57:25.680 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-01-24T16:57:26.042 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-waitress. 2023-01-24T16:57:26.059 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../040-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-01-24T16:57:26.120 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-01-24T16:57:26.445 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-tempita. 2023-01-24T16:57:26.461 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../041-python3-tempita_0.5.2-6_all.deb ... 2023-01-24T16:57:26.503 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-01-24T16:57:26.798 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-paste. 2023-01-24T16:57:26.816 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../042-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-01-24T16:57:26.856 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-01-24T16:57:27.167 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-01-24T16:57:27.183 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../043-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-01-24T16:57:27.225 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-01-24T16:57:27.528 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-pastedeploy. 2023-01-24T16:57:27.547 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../044-python3-pastedeploy_2.1.0-1_all.deb ... 2023-01-24T16:57:27.587 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-01-24T16:57:27.840 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-webtest. 2023-01-24T16:57:27.859 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../045-python3-webtest_2.0.32-2_all.deb ... 2023-01-24T16:57:27.898 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-01-24T16:57:28.201 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-pecan. 2023-01-24T16:57:28.217 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../046-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-01-24T16:57:28.284 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-01-24T16:57:28.672 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-werkzeug. 2023-01-24T16:57:28.688 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../047-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2023-01-24T16:57:28.730 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2023-01-24T16:57:29.025 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-01-24T16:57:29.043 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../048-libsqlite3-mod-ceph_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:29.108 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libsqlite3-mod-ceph (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:29.428 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-mgr. 2023-01-24T16:57:29.444 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../049-ceph-mgr_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:29.486 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-mgr (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:30.015 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-mon. 2023-01-24T16:57:30.032 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../050-ceph-mon_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:30.073 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-mon (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:31.477 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-osd. 2023-01-24T16:57:31.495 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../051-ceph-osd_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:31.951 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-osd (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:34.830 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph. 2023-01-24T16:57:34.848 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../052-ceph_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:34.879 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:35.182 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-fuse. 2023-01-24T16:57:35.198 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../053-ceph-fuse_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:35.240 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-fuse (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:35.694 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-mds. 2023-01-24T16:57:35.709 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../054-ceph-mds_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:35.752 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-mds (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:36.265 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package cephadm. 2023-01-24T16:57:36.281 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../055-cephadm_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:36.348 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking cephadm (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:36.659 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-jinja2. 2023-01-24T16:57:36.676 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../056-python3-jinja2_2.10.1-2_all.deb ... 2023-01-24T16:57:36.717 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-01-24T16:57:37.029 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-01-24T16:57:37.045 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../057-ceph-mgr-cephadm_17.2.5-727-g49f8fb05-1focal_all.deb ... 2023-01-24T16:57:37.155 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-mgr-cephadm (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:37.491 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-repoze.lru. 2023-01-24T16:57:37.510 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../058-python3-repoze.lru_0.7-2_all.deb ... 2023-01-24T16:57:37.549 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-01-24T16:57:37.785 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libjs-underscore. 2023-01-24T16:57:37.803 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../059-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-01-24T16:57:37.843 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-01-24T16:57:38.129 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-01-24T16:57:38.147 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../060-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-01-24T16:57:38.187 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-01-24T16:57:38.515 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-routes. 2023-01-24T16:57:38.532 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../061-python3-routes_2.4.1-2_all.deb ... 2023-01-24T16:57:38.573 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-01-24T16:57:38.902 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-01-24T16:57:38.921 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../062-ceph-mgr-dashboard_17.2.5-727-g49f8fb05-1focal_all.deb ... 2023-01-24T16:57:38.960 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-mgr-dashboard (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:39.406 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-decorator. 2023-01-24T16:57:39.423 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../063-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-01-24T16:57:39.463 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-01-24T16:57:39.775 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-01-24T16:57:39.795 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../064-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-01-24T16:57:39.833 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-01-24T16:57:40.128 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-scipy. 2023-01-24T16:57:40.147 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../065-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-01-24T16:57:40.203 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-01-24T16:57:41.496 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-01-24T16:57:41.512 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../066-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-01-24T16:57:41.554 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-01-24T16:57:42.091 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-joblib. 2023-01-24T16:57:42.111 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../067-python3-joblib_0.14.0-3_all.deb ... 2023-01-24T16:57:42.149 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-01-24T16:57:42.503 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-sklearn. 2023-01-24T16:57:42.520 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../068-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-01-24T16:57:42.561 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-01-24T16:57:43.335 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-01-24T16:57:43.355 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../069-ceph-mgr-diskprediction-local_17.2.5-727-g49f8fb05-1focal_all.deb ... 2023-01-24T16:57:43.393 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:44.529 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-cachetools. 2023-01-24T16:57:44.546 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../070-python3-cachetools_4.0.0-1_all.deb ... 2023-01-24T16:57:44.587 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-01-24T16:57:44.899 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-pyasn1. 2023-01-24T16:57:44.916 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../071-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-01-24T16:57:44.957 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-01-24T16:57:45.295 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-01-24T16:57:45.312 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../072-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-01-24T16:57:45.353 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-01-24T16:57:45.665 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-rsa. 2023-01-24T16:57:45.681 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../073-python3-rsa_4.0-3_all.deb ... 2023-01-24T16:57:45.723 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-rsa (4.0-3) ... 2023-01-24T16:57:46.036 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-google-auth. 2023-01-24T16:57:46.055 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../074-python3-google-auth_1.5.1-2_all.deb ... 2023-01-24T16:57:46.094 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-01-24T16:57:46.415 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-blinker. 2023-01-24T16:57:46.433 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../075-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-01-24T16:57:46.474 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-01-24T16:57:46.811 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-oauthlib. 2023-01-24T16:57:46.827 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../076-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-01-24T16:57:46.869 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-01-24T16:57:47.197 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-01-24T16:57:47.216 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../077-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-01-24T16:57:47.256 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-01-24T16:57:47.918 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-websocket. 2023-01-24T16:57:47.934 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../078-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-01-24T16:57:47.975 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-01-24T16:57:48.296 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-kubernetes. 2023-01-24T16:57:48.313 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../079-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-01-24T16:57:48.354 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-01-24T16:57:48.926 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-01-24T16:57:48.944 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../080-ceph-mgr-k8sevents_17.2.5-727-g49f8fb05-1focal_all.deb ... 2023-01-24T16:57:48.984 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-mgr-k8sevents (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:49.423 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libonig5:amd64. 2023-01-24T16:57:49.440 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../081-libonig5_6.9.4-1_amd64.deb ... 2023-01-24T16:57:49.481 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-01-24T16:57:49.826 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libjq1:amd64. 2023-01-24T16:57:49.842 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../082-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-01-24T16:57:49.884 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-01-24T16:57:50.162 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package jq. 2023-01-24T16:57:50.180 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../083-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-01-24T16:57:50.212 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-01-24T16:57:50.465 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package socat. 2023-01-24T16:57:50.483 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../084-socat_1.7.3.3-2_amd64.deb ... 2023-01-24T16:57:50.523 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking socat (1.7.3.3-2) ... 2023-01-24T16:57:50.868 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package xmlstarlet. 2023-01-24T16:57:50.887 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-01-24T16:57:50.935 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-01-24T16:57:51.267 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-test. 2023-01-24T16:57:51.287 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../086-ceph-test_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:51.320 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-test (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:56.481 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package ceph-volume. 2023-01-24T16:57:56.498 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../087-ceph-volume_17.2.5-727-g49f8fb05-1focal_all.deb ... 2023-01-24T16:57:56.560 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking ceph-volume (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:56.892 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package fonts-mathjax. 2023-01-24T16:57:56.909 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../088-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-01-24T16:57:56.950 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-01-24T16:57:57.488 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libcephfs-dev. 2023-01-24T16:57:57.506 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../089-libcephfs-dev_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:57:57.546 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libcephfs-dev (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:57:57.942 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package nvme-cli. 2023-01-24T16:57:57.958 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../090-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-01-24T16:57:58.017 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-01-24T16:57:58.397 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libjs-mathjax. 2023-01-24T16:57:58.416 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../091-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-01-24T16:57:58.455 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-01-24T16:58:00.108 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python-natsort-doc. 2023-01-24T16:58:00.127 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../092-python-natsort-doc_7.0.1-1_all.deb ... 2023-01-24T16:58:00.166 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-01-24T16:58:00.486 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-atomicwrites. 2023-01-24T16:58:00.503 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../093-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-01-24T16:58:00.544 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-01-24T16:58:00.880 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-attr. 2023-01-24T16:58:00.899 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../094-python3-attr_19.3.0-2_all.deb ... 2023-01-24T16:58:00.938 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-01-24T16:58:01.267 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-webencodings. 2023-01-24T16:58:01.285 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../095-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-01-24T16:58:01.324 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-01-24T16:58:01.653 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-html5lib. 2023-01-24T16:58:01.670 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../096-python3-html5lib_1.0.1-2_all.deb ... 2023-01-24T16:58:01.702 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-01-24T16:58:02.031 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-01-24T16:58:02.051 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../097-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-01-24T16:58:02.089 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-01-24T16:58:02.559 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-packaging. 2023-01-24T16:58:02.579 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../098-python3-packaging_20.3-1_all.deb ... 2023-01-24T16:58:02.617 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-packaging (20.3-1) ... 2023-01-24T16:58:02.912 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-pastescript. 2023-01-24T16:58:02.929 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2023-01-24T16:58:02.989 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-01-24T16:58:03.307 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-pluggy. 2023-01-24T16:58:03.324 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2023-01-24T16:58:03.365 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-01-24T16:58:03.677 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-psutil. 2023-01-24T16:58:03.694 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-01-24T16:58:03.735 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-01-24T16:58:04.097 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-py. 2023-01-24T16:58:04.114 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-01-24T16:58:04.155 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-01-24T16:58:04.475 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-pyinotify. 2023-01-24T16:58:04.493 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-01-24T16:58:04.533 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-01-24T16:58:04.930 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-wcwidth. 2023-01-24T16:58:04.949 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-01-24T16:58:04.988 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-01-24T16:58:05.291 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-pytest. 2023-01-24T16:58:05.309 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2023-01-24T16:58:05.350 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-01-24T16:58:05.669 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package python3-simplejson. 2023-01-24T16:58:05.688 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-01-24T16:58:05.727 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-01-24T16:58:05.989 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-01-24T16:58:06.007 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-01-24T16:58:06.048 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-01-24T16:58:06.785 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package radosgw. 2023-01-24T16:58:06.801 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../108-radosgw_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:58:06.843 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking radosgw (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:08.555 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package rbd-fuse. 2023-01-24T16:58:08.573 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../109-rbd-fuse_17.2.5-727-g49f8fb05-1focal_amd64.deb ... 2023-01-24T16:58:08.638 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking rbd-fuse (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:08.941 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-attr (19.3.0-2) ... 2023-01-24T16:58:09.235 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-01-24T16:58:09.361 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-01-24T16:58:09.513 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-01-24T16:58:09.715 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up cephadm (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:10.343 INFO:teuthology.orchestra.run.smithi103.stdout:Adding system user cephadm....done 2023-01-24T16:58:10.504 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-01-24T16:58:10.631 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-01-24T16:58:10.982 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-01-24T16:58:11.300 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-01-24T16:58:11.418 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-01-24T16:58:11.536 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-01-24T16:58:11.871 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-01-24T16:58:12.323 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-01-24T16:58:12.641 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-01-24T16:58:12.934 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-01-24T16:58:13.218 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-01-24T16:58:13.519 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-ceph-argparse (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:13.812 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-01-24T16:58:13.938 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-01-24T16:58:14.225 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-01-24T16:58:14.507 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-01-24T16:58:14.633 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-01-24T16:58:15.035 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-01-24T16:58:15.336 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-01-24T16:58:15.612 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-01-24T16:58:15.955 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-01-24T16:58:16.231 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-01-24T16:58:16.558 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-01-24T16:58:16.875 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-packaging (20.3-1) ... 2023-01-24T16:58:17.177 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-01-24T16:58:17.303 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-01-24T16:58:17.454 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-01-24T16:58:17.738 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-01-24T16:58:18.132 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2023-01-24T16:58:18.725 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-01-24T16:58:19.093 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up socat (1.7.3.3-2) ... 2023-01-24T16:58:19.220 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-ceph-common (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:19.530 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-24T16:58:19.664 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-01-24T16:58:19.991 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-01-24T16:58:20.117 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-01-24T16:58:20.243 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-01-24T16:58:20.519 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-01-24T16:58:20.829 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-24T16:58:20.955 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-01-24T16:58:21.081 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-01-24T16:58:21.391 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-01-24T16:58:21.709 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-01-24T16:58:21.993 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-01-24T16:58:22.119 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-01-24T16:58:22.404 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-01-24T16:58:22.697 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-01-24T16:58:22.989 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-01-24T16:58:23.447 INFO:teuthology.orchestra.run.smithi103.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-01-24T16:58:23.701 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-01-24T16:58:23.827 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-01-24T16:58:24.169 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-01-24T16:58:24.523 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-01-24T16:58:24.672 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-01-24T16:58:24.798 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-01-24T16:58:25.100 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-01-24T16:58:25.376 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up librados2 (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:25.527 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-01-24T16:58:27.000 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libsqlite3-mod-ceph (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:27.176 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-01-24T16:58:27.285 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-01-24T16:58:27.662 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-01-24T16:58:27.972 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-01-24T16:58:28.273 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-24T16:58:28.399 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-01-24T16:58:28.742 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libcephfs2 (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:28.868 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-01-24T16:58:29.178 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libradosstriper1 (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:29.304 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-01-24T16:58:29.639 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-01-24T16:58:30.695 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up librbd1 (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:30.821 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-01-24T16:58:30.955 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-01-24T16:58:31.106 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-01-24T16:58:31.233 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-rsa (4.0-3) ... 2023-01-24T16:58:31.542 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-01-24T16:58:31.661 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-01-24T16:58:31.970 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-mgr-modules-core (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:32.096 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-fuse (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:32.322 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-01-24T16:58:32.322 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-01-24T16:58:32.765 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libcephfs-dev (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:32.976 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-01-24T16:58:33.278 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-01-24T16:58:33.563 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-01-24T16:58:33.907 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-rados (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:34.042 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up librgw2 (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:34.168 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-rbd (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:34.302 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up rbd-fuse (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:34.412 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-routes (2.4.1-2) ... 2023-01-24T16:58:34.696 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-01-24T16:58:34.998 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-rgw (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:35.149 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-cephfs (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:35.276 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-01-24T16:58:36.192 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-common (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:36.720 INFO:teuthology.orchestra.run.smithi103.stdout:Adding group ceph....done 2023-01-24T16:58:37.096 INFO:teuthology.orchestra.run.smithi103.stdout:Adding system user ceph....done 2023-01-24T16:58:37.273 INFO:teuthology.orchestra.run.smithi103.stdout:Setting system user ceph properties....done 2023-01-24T16:58:37.295 INFO:teuthology.orchestra.run.smithi103.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-01-24T16:58:37.442 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-01-24T16:58:37.761 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-01-24T16:58:38.202 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-test (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:38.353 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up radosgw (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:38.627 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-01-24T16:58:38.627 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-01-24T16:58:39.281 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-base (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:39.663 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-01-24T16:58:40.101 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-mds (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:40.318 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-01-24T16:58:40.319 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-01-24T16:58:40.752 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-mgr (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:40.983 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-01-24T16:58:40.984 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-01-24T16:58:41.429 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-osd (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:41.954 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-01-24T16:58:41.954 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-01-24T16:58:42.389 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-mgr-k8sevents (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:42.515 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-mgr-diskprediction-local (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:42.634 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-mon (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:42.845 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-01-24T16:58:42.846 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-01-24T16:58:43.293 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-mgr-cephadm (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:43.411 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:43.545 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-mgr-dashboard (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:43.688 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up ceph-volume (17.2.5-727-g49f8fb05-1focal) ... 2023-01-24T16:58:43.815 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-01-24T16:58:43.984 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-01-24T16:58:44.319 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-01-24T16:58:47.164 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-01-24T16:58:47.747 INFO:teuthology.orchestra.run.smithi103.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-01-24T16:58:47.750 DEBUG:teuthology.parallel:result is None 2023-01-24T16:58:47.751 DEBUG:teuthology.orchestra.run.smithi103:> sudo lsb_release -is 2023-01-24T16:58:47.798 INFO:teuthology.orchestra.run.smithi103.stdout:Ubuntu 2023-01-24T16:58:47.809 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-24T16:58:47.810 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=49f8fb05584886826e8eade75f7105fba754560c 2023-01-24T16:58:47.960 DEBUG:teuthology.orchestra.run.smithi103:> dpkg-query -W -f '${Version}' ceph 2023-01-24T16:58:47.985 INFO:teuthology.orchestra.run.smithi103.stdout:17.2.5-727-g49f8fb05-1focal 2023-01-24T16:58:47.986 INFO:teuthology.packaging:The installed version of ceph is 17.2.5-727-g49f8fb05-1focal 2023-01-24T16:58:47.986 INFO:teuthology.task.install:The correct ceph version 17.2.5-727-g49f8fb05-1focal is installed. 2023-01-24T16:58:47.987 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-01-24T16:58:47.987 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:47.988 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-24T16:58:48.044 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-01-24T16:58:48.045 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:48.045 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2023-01-24T16:58:48.055 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-24T16:58:48.065 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-01-24T16:58:48.065 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:48.066 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-24T16:58:48.078 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-24T16:58:48.136 INFO:teuthology.run_tasks:Running task ceph... 2023-01-24T16:58:48.199 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-01-24T16:58:48.199 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /var/log/ceph 2023-01-24T16:58:48.214 INFO:tasks.ceph:Disabling ceph logrotate... 2023-01-24T16:58:48.214 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/logrotate.d/ceph 2023-01-24T16:58:48.274 INFO:tasks.ceph:Creating extra log directories... 2023-01-24T16:58:48.275 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-01-24T16:58:48.295 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-01-24T16:58:48.296 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt kmip ca path': '/etc/ceph/kmiproot.crt', 'rgw crypt kmip client cert': '/etc/ceph/kmip-client.crt', 'rgw crypt kmip client key': '/etc/ceph/kmip-client.key', 'rgw crypt kmip kms key template': 'pykmip-$keyid', 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'kmip', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-01-24T16:58:48.296 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135469', 'branch': 'wip-yuri7-testing-2023-01-23-1532-quincy', 'description': 'rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}}', 'email': 'yweinstein@radhat.com', 'first_in_suite': False, 'job_id': '7135469', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri7-testing-2023-01-23-1532-quincy'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt kmip ca path': '/etc/ceph/kmiproot.crt', 'rgw crypt kmip client cert': '/etc/ceph/kmip-client.crt', 'rgw crypt kmip client key': '/etc/ceph/kmip-client.key', 'rgw crypt kmip kms key template': 'pykmip-$keyid', 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'kmip', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '49f8fb05584886826e8eade75f7105fba754560c', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'}}, 'rgw': {'client.0': {'use-pykmip-role': 'client.0'}}, 'workunit': {'branch': 'wip-yuri7-testing-2023-01-23-1532-quincy', 'sha1': '49f8fb05584886826e8eade75f7105fba754560c'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '49f8fb05584886826e8eade75f7105fba754560c', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-yuri7-testing-2023-01-23-1532-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_49f8fb05584886826e8eade75f7105fba754560c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '49f8fb05584886826e8eade75f7105fba754560c', 'targets': {'smithi103.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'openssl_keys': {'kmip-client': {'ca': 'kmiproot', 'client': 'client.0', 'cn': 'rgw-client'}, 'kmip-server': {'ca': 'kmiproot', 'client': 'client.0'}, 'kmiproot': {'client': 'client.0', 'cn': 'kmiproot', 'key-type': 'rsa:4096'}}}, {'exec': {'client.0': ['chmod 644 /home/ubuntu/cephtest/ca/kmip-client.key']}}, {'pykmip': {'client.0': {'clientca': 'kmiproot', 'clientcert': 'kmip-client', 'secrets': [{'name': 'pykmip-my-key-1'}, {'name': 'pykmip-my-key-2'}], 'servercert': 'kmip-server'}}}, {'rgw': {'client.0': None}}, {'s3tests': {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-quincy', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8e1bba12e584b7ae912e92e94afe9c075ab51fa3', 'timestamp': '2023-01-24_16:04:03', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.17735'} 2023-01-24T16:58:48.297 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-01-24T16:58:48.303 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m0777 -- /var/run/ceph 2023-01-24T16:58:48.362 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:48.362 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2023-01-24T16:58:48.367 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-01-24T16:58:48.368 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2023-01-24T16:58:48.372 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-01-24T16:58:48.373 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T16:58:48.373 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 359 Links: 1 2023-01-24T16:58:48.373 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T16:58:48.373 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2023-01-24 16:55:31.669773819 +0000 2023-01-24T16:58:48.373 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2023-01-24 16:55:31.409779801 +0000 2023-01-24T16:58:48.373 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2023-01-24 16:55:31.409779801 +0000 2023-01-24T16:58:48.374 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2023-01-24T16:58:48.374 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-01-24T16:58:48.384 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2023-01-24T16:58:48.384 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2023-01-24T16:58:48.384 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000396652 s, 1.3 MB/s 2023-01-24T16:58:48.385 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-01-24T16:58:48.437 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2023-01-24T16:58:48.444 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-01-24T16:58:48.444 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T16:58:48.445 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 367 Links: 1 2023-01-24T16:58:48.445 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T16:58:48.445 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2023-01-24 16:55:32.161762497 +0000 2023-01-24T16:58:48.445 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2023-01-24 16:55:31.901768479 +0000 2023-01-24T16:58:48.445 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2023-01-24 16:55:31.901768479 +0000 2023-01-24T16:58:48.446 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2023-01-24T16:58:48.447 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-01-24T16:58:48.461 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2023-01-24T16:58:48.461 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2023-01-24T16:58:48.461 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000406232 s, 1.3 MB/s 2023-01-24T16:58:48.462 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-01-24T16:58:48.518 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2023-01-24T16:58:48.526 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-01-24T16:58:48.526 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T16:58:48.526 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 374 Links: 1 2023-01-24T16:58:48.526 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T16:58:48.527 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2023-01-24 16:55:32.653751174 +0000 2023-01-24T16:58:48.527 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2023-01-24 16:55:32.385757341 +0000 2023-01-24T16:58:48.527 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2023-01-24 16:55:32.385757341 +0000 2023-01-24T16:58:48.527 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2023-01-24T16:58:48.528 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-01-24T16:58:48.540 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2023-01-24T16:58:48.541 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2023-01-24T16:58:48.541 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000324837 s, 1.6 MB/s 2023-01-24T16:58:48.542 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-01-24T16:58:48.593 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2023-01-24T16:58:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-01-24T16:58:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-24T16:58:48.601 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 377 Links: 1 2023-01-24T16:58:48.601 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-24T16:58:48.602 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2023-01-24 16:55:33.161739485 +0000 2023-01-24T16:58:48.602 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2023-01-24 16:55:32.865746296 +0000 2023-01-24T16:58:48.602 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2023-01-24 16:55:32.865746296 +0000 2023-01-24T16:58:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: - 2023-01-24T16:58:48.602 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-01-24T16:58:48.615 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2023-01-24T16:58:48.615 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2023-01-24T16:58:48.615 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000414695 s, 1.2 MB/s 2023-01-24T16:58:48.620 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-01-24T16:58:48.669 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T16:58:48.670 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-01-24T16:58:48.670 INFO:tasks.ceph:Generating config... 2023-01-24T16:58:48.671 INFO:tasks.ceph:[client] debug rgw = 20 2023-01-24T16:58:48.672 INFO:tasks.ceph:[client] rgw crypt kmip ca path = /etc/ceph/kmiproot.crt 2023-01-24T16:58:48.672 INFO:tasks.ceph:[client] rgw crypt kmip client cert = /etc/ceph/kmip-client.crt 2023-01-24T16:58:48.672 INFO:tasks.ceph:[client] rgw crypt kmip client key = /etc/ceph/kmip-client.key 2023-01-24T16:58:48.672 INFO:tasks.ceph:[client] rgw crypt kmip kms key template = pykmip-$keyid 2023-01-24T16:58:48.673 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-01-24T16:58:48.673 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = kmip 2023-01-24T16:58:48.673 INFO:tasks.ceph:[client] setgroup = ceph 2023-01-24T16:58:48.673 INFO:tasks.ceph:[client] setuser = ceph 2023-01-24T16:58:48.674 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-01-24T16:58:48.674 INFO:tasks.ceph:[mgr] debug ms = 1 2023-01-24T16:58:48.674 INFO:tasks.ceph:[mon] debug mon = 20 2023-01-24T16:58:48.675 INFO:tasks.ceph:[mon] debug ms = 1 2023-01-24T16:58:48.675 INFO:tasks.ceph:[mon] debug paxos = 20 2023-01-24T16:58:48.675 INFO:tasks.ceph:[osd] debug ms = 1 2023-01-24T16:58:48.675 INFO:tasks.ceph:[osd] debug osd = 20 2023-01-24T16:58:48.676 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-01-24T16:58:48.676 INFO:tasks.ceph:Setting up mon.a... 2023-01-24T16:58:48.676 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-01-24T16:58:48.711 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/ceph.keyring 2023-01-24T16:58:48.714 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-01-24T16:58:48.744 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T16:58:48.801 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.103')] 2023-01-24T16:58:48.802 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.103'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt kmip ca path': '/etc/ceph/kmiproot.crt', 'rgw crypt kmip client cert': '/etc/ceph/kmip-client.crt', 'rgw crypt kmip client key': '/etc/ceph/kmip-client.key', 'rgw crypt kmip kms key template': 'pykmip-$keyid', 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'kmip', 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2023-01-24T16:58:48.803 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:48.803 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-01-24T16:58:48.810 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.103 --print /home/ubuntu/cephtest/ceph.monmap 2023-01-24T16:58:48.844 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setuser ceph since I am not root 2023-01-24T16:58:48.844 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T16:58:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-01-24T16:58:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:monmaptool: generated fsid bf3b1473-4375-4731-8219-58951a76fe3f 2023-01-24T16:58:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:setting min_mon_release = octopus 2023-01-24T16:58:48.845 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 0 2023-01-24T16:58:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:fsid bf3b1473-4375-4731-8219-58951a76fe3f 2023-01-24T16:58:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:last_changed 2023-01-24T16:58:48.839041+0000 2023-01-24T16:58:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:created 2023-01-24T16:58:48.839041+0000 2023-01-24T16:58:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:min_mon_release 15 (octopus) 2023-01-24T16:58:48.846 INFO:teuthology.orchestra.run.smithi103.stdout:election_strategy: 1 2023-01-24T16:58:48.847 INFO:teuthology.orchestra.run.smithi103.stdout:0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.a 2023-01-24T16:58:48.847 INFO:teuthology.orchestra.run.smithi103.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-01-24T16:58:48.848 DEBUG:teuthology.orchestra.run.smithi103:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-01-24T16:58:48.855 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID bf3b1473-4375-4731-8219-58951a76fe3f... 2023-01-24T16:58:48.856 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-01-24T16:58:48.930 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2023-01-24T16:58:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: chdir = "" 2023-01-24T16:58:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-01-24T16:58:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: auth supported = cephx 2023-01-24T16:58:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: filestore xattr use omap = true 2023-01-24T16:58:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: mon clock drift allowed = 1.000 2023-01-24T16:58:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: osd crush chooseleaf type = 0 2023-01-24T16:58:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: auth debug = true 2023-01-24T16:58:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on old message = true 2023-01-24T16:58:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: ms die on bug = true 2023-01-24T16:58:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: mon max pg per osd = 10000 # >= luminous 2023-01-24T16:58:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: mon pg warn max object skew = 0 2023-01-24T16:58:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: # disable pg_autoscaler by default for new pools 2023-01-24T16:58:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: osd_pool_default_pg_autoscale_mode = off 2023-01-24T16:58:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default size = 2 2023-01-24T16:58:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow primary affinity = true 2023-01-24T16:58:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd allow pg remap = true 2023-01-24T16:58:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on legacy crush tunables = false 2023-01-24T16:58:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on crush straw calc version zero = false 2023-01-24T16:58:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on no sortbitwise = false 2023-01-24T16:58:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on osd down out interval zero = false 2023-01-24T16:58:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: mon warn on too few osds = false 2023-01-24T16:58:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-01-24T16:58:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_pool_no_redundancy = false 2023-01-24T16:58:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: mon_allow_pool_size_one = true 2023-01-24T16:58:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-01-24T16:58:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: osd default data pool replay window = 5 2023-01-24T16:58:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: mon allow pool delete = true 2023-01-24T16:58:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: mon cluster log file level = debug 2023-01-24T16:58:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: debug asserts on shutdown = true 2023-01-24T16:58:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: mon health detail to clog = false 2023-01-24T16:58:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: mon host = 172.21.15.103 2023-01-24T16:58:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = bf3b1473-4375-4731-8219-58951a76fe3f 2023-01-24T16:58:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.940 INFO:teuthology.orchestra.run.smithi103.stdout:[osd] 2023-01-24T16:58:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: osd journal size = 100 2023-01-24T16:58:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub load threshold = 5.0 2023-01-24T16:58:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: osd scrub max interval = 600 2023-01-24T16:58:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: osd recover clone overlap = true 2023-01-24T16:58:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: osd recovery max chunk = 1048576 2023-01-24T16:58:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug shutdown = true 2023-01-24T16:58:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug op order = true 2023-01-24T16:58:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug verify stray on activate = true 2023-01-24T16:58:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: osd open classes on start = true 2023-01-24T16:58:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug pg log writeout = true 2023-01-24T16:58:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: osd deep scrub update digest min age = 30 2023-01-24T16:58:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: osd map max advance = 10 2023-01-24T16:58:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: journal zero on create = true 2023-01-24T16:58:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: filestore ondisk finisher threads = 3 2023-01-24T16:58:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: filestore apply finisher threads = 3 2023-01-24T16:58:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: bdev debug aio = true 2023-01-24T16:58:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: osd debug misdirected ops = true 2023-01-24T16:58:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2023-01-24T16:58:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: debug osd = 20 2023-01-24T16:58:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: osd shutdown pgref assert = True 2023-01-24T16:58:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.946 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr] 2023-01-24T16:58:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2023-01-24T16:58:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: debug mgr = 20 2023-01-24T16:58:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2023-01-24T16:58:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2023-01-24T16:58:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2023-01-24T16:58:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2023-01-24T16:58:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: mgr/telemetry/nag = false 2023-01-24T16:58:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.948 INFO:teuthology.orchestra.run.smithi103.stdout:[mon] 2023-01-24T16:58:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: debug ms = 1 2023-01-24T16:58:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: debug mon = 20 2023-01-24T16:58:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: debug paxos = 20 2023-01-24T16:58:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: debug auth = 20 2023-01-24T16:58:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: mon data avail warn = 5 2023-01-24T16:58:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: mon mgr mkfs grace = 240 2023-01-24T16:58:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min pgs per osd = 4 2023-01-24T16:58:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd reporter subtree level = osd 2023-01-24T16:58:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: mon osd prime pg temp = true 2023-01-24T16:58:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: mon reweight min bytes per osd = 10 2023-01-24T16:58:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-01-24T16:58:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: auth mon ticket ttl = 660 # 11m 2023-01-24T16:58:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: auth service ticket ttl = 240 # 4m 2023-01-24T16:58:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: # don't complain about insecure global_id in the test suite 2023-01-24T16:58:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-01-24T16:58:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-01-24T16:58:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: # 1m isn't quite enough 2023-01-24T16:58:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: mon_down_mkfs_grace = 2m 2023-01-24T16:58:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: mon_warn_on_filestore_osds = false 2023-01-24T16:58:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:48.952 INFO:teuthology.orchestra.run.smithi103.stdout:[client] 2023-01-24T16:58:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: rgw cache enabled = true 2023-01-24T16:58:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable ops log = true 2023-01-24T16:58:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: rgw enable usage log = true 2023-01-24T16:58:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-01-24T16:58:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-01-24T16:58:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: debug rgw = 20 2023-01-24T16:58:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: rgw crypt kmip ca path = /etc/ceph/kmiproot.crt 2023-01-24T16:58:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: rgw crypt kmip client cert = /etc/ceph/kmip-client.crt 2023-01-24T16:58:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: rgw crypt kmip client key = /etc/ceph/kmip-client.key 2023-01-24T16:58:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: rgw crypt kmip kms key template = pykmip-$keyid 2023-01-24T16:58:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: rgw crypt require ssl = False 2023-01-24T16:58:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: rgw crypt s3 kms backend = kmip 2023-01-24T16:58:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: setgroup = ceph 2023-01-24T16:58:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: setuser = ceph 2023-01-24T16:58:48.955 INFO:teuthology.orchestra.run.smithi103.stdout:[mon.a] 2023-01-24T16:58:48.956 INFO:tasks.ceph:Creating admin key on mon.a... 2023-01-24T16:58:48.957 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-01-24T16:58:49.004 INFO:tasks.ceph:Copying monmap to all nodes... 2023-01-24T16:58:49.005 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:49.005 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-01-24T16:58:49.012 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:49.012 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-01-24T16:58:49.059 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi103.front.sepia.ceph.com 2023-01-24T16:58:49.060 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:49.060 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.keyring 2023-01-24T16:58:49.060 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-24T16:58:49.086 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:49.086 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-01-24T16:58:49.091 INFO:tasks.ceph:Setting up mon nodes... 2023-01-24T16:58:49.092 INFO:tasks.ceph:Setting up mgr nodes... 2023-01-24T16:58:49.092 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-01-24T16:58:49.176 INFO:teuthology.orchestra.run.smithi103.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-01-24T16:58:49.179 INFO:tasks.ceph:Setting up mds nodes... 2023-01-24T16:58:49.179 INFO:tasks.ceph_client:Setting up client nodes... 2023-01-24T16:58:49.180 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-01-24T16:58:49.213 INFO:teuthology.orchestra.run.smithi103.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-01-24T16:58:49.225 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-01-24T16:58:49.226 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi103.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-01-24T16:58:49.226 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-01-24T16:58:49.238 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T16:58:49.238 INFO:tasks.ceph:role: osd.0 2023-01-24T16:58:49.239 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com 2023-01-24T16:58:49.239 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-01-24T16:58:49.806 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T16:58:49.807 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T16:58:49.807 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T16:58:49.807 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 2023-01-24T16:58:49.807 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T16:58:49.807 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2023-01-24T16:58:49.808 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T16:58:49.808 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T16:58:49.808 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T16:58:49.808 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T16:58:49.809 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2023-01-24T16:58:49.810 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-01-24T16:58:49.908 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-01-24T16:58:49.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T16:58:49.922 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: /sbin/restorecon: command not found 2023-01-24T16:58:49.922 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-01-24T16:58:49.979 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T16:58:49.979 INFO:tasks.ceph:role: osd.1 2023-01-24T16:58:49.980 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com 2023-01-24T16:58:49.980 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-01-24T16:58:50.484 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T16:58:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T16:58:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T16:58:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 2023-01-24T16:58:50.485 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T16:58:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2023-01-24T16:58:50.486 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T16:58:50.486 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T16:58:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T16:58:50.486 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T16:58:50.487 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2023-01-24T16:58:50.488 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-01-24T16:58:50.525 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-01-24T16:58:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T16:58:50.535 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: /sbin/restorecon: command not found 2023-01-24T16:58:50.535 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-01-24T16:58:50.593 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-01-24T16:58:50.594 INFO:tasks.ceph:role: osd.2 2023-01-24T16:58:50.594 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com 2023-01-24T16:58:50.595 DEBUG:teuthology.orchestra.run.smithi103:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-01-24T16:58:51.072 INFO:teuthology.orchestra.run.smithi103.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-01-24T16:58:51.072 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-24T16:58:51.072 INFO:teuthology.orchestra.run.smithi103.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-24T16:58:51.073 INFO:teuthology.orchestra.run.smithi103.stdout: = reflink=1 2023-01-24T16:58:51.073 INFO:teuthology.orchestra.run.smithi103.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-24T16:58:51.073 INFO:teuthology.orchestra.run.smithi103.stdout: = sunit=0 swidth=0 blks 2023-01-24T16:58:51.073 INFO:teuthology.orchestra.run.smithi103.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-24T16:58:51.073 INFO:teuthology.orchestra.run.smithi103.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-24T16:58:51.074 INFO:teuthology.orchestra.run.smithi103.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-24T16:58:51.074 INFO:teuthology.orchestra.run.smithi103.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-24T16:58:51.075 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi103.front.sepia.ceph.com -o noatime 2023-01-24T16:58:51.075 DEBUG:teuthology.orchestra.run.smithi103:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-01-24T16:58:51.137 DEBUG:teuthology.orchestra.run.smithi103:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-01-24T16:58:51.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T16:58:51.150 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: /sbin/restorecon: command not found 2023-01-24T16:58:51.151 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T16:58:51.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:51.212+0000 7f68008223c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-01-24T16:58:51.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:51.216+0000 7f68008223c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-01-24T16:58:51.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:51.216+0000 7f68008223c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-24T16:58:51.226 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:51.216+0000 7f68008223c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-24T16:58:51.226 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:51.216+0000 7f68008223c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-24T16:58:51.229 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:51.216+0000 7f68008223c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-01-24T16:58:53.426 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-01-24T16:58:53.439 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T16:58:53.480 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:53.468+0000 7f351df663c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-01-24T16:58:53.480 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:53.468+0000 7f351df663c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-01-24T16:58:53.480 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:53.468+0000 7f351df663c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-24T16:58:53.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:53.468+0000 7f351df663c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-24T16:58:53.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:53.468+0000 7f351df663c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-24T16:58:53.481 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:53.468+0000 7f351df663c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-01-24T16:58:55.690 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-01-24T16:58:55.750 DEBUG:teuthology.orchestra.run.smithi103:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-24T16:58:55.790 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:55.780+0000 7fd3b70a73c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-01-24T16:58:55.790 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:55.780+0000 7fd3b70a73c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-01-24T16:58:55.791 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:55.780+0000 7fd3b70a73c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-24T16:58:55.791 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:55.780+0000 7fd3b70a73c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-24T16:58:55.791 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:55.780+0000 7fd3b70a73c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-24T16:58:55.791 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T16:58:55.780+0000 7fd3b70a73c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-01-24T16:58:57.980 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-01-24T16:58:57.995 INFO:tasks.ceph:Reading keys from all nodes... 2023-01-24T16:58:57.995 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:57.996 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-01-24T16:58:58.051 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:58.052 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-01-24T16:58:58.064 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:58.064 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-01-24T16:58:58.076 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:58.077 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-01-24T16:58:58.088 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:58.088 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-01-24T16:58:58.135 INFO:tasks.ceph:Adding keys to all mons... 2023-01-24T16:58:58.136 DEBUG:teuthology.orchestra.run.smithi103:> sudo tee -a /etc/ceph/ceph.keyring 2023-01-24T16:58:58.147 INFO:teuthology.orchestra.run.smithi103.stdout:[mgr.x] 2023-01-24T16:58:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBJDtBjKh05ChAAT3wk/eSc+uN/3LMrQW6bSw== 2023-01-24T16:58:58.148 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.0] 2023-01-24T16:58:58.148 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBLDtBjUBLxDBAA9IeHo5nWzB71uK4FVcr6mw== 2023-01-24T16:58:58.149 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.1] 2023-01-24T16:58:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBNDtBj8jpEHBAARbdYReC/3oGJYk8rwSqFiA== 2023-01-24T16:58:58.149 INFO:teuthology.orchestra.run.smithi103.stdout:[osd.2] 2023-01-24T16:58:58.149 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBPDtBjxQDCLhAAJi6GHwogUu4BDTUC04WiGg== 2023-01-24T16:58:58.149 INFO:teuthology.orchestra.run.smithi103.stdout:[client.0] 2023-01-24T16:58:58.150 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQBJDtBj9FtsDBAAWiFJmv7CLwIMcsQJZEjl5w== 2023-01-24T16:58:58.151 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-24T16:58:58.192 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T16:58:58.277 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T16:58:58.320 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-24T16:58:58.366 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-24T16:58:58.406 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-01-24T16:58:58.407 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-01-24T16:58:58.420 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-24T16:58:58.471 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-01-24T16:58:58.484 DEBUG:teuthology.orchestra.run.smithi103:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-01-24T16:58:58.489 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-01-24T16:58:58.490 INFO:tasks.ceph.mon.a:Restarting daemon 2023-01-24T16:58:58.490 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-01-24T16:58:58.535 INFO:tasks.ceph.mon.a:Started 2023-01-24T16:58:58.535 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-01-24T16:58:58.535 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-01-24T16:58:58.536 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-01-24T16:58:58.539 INFO:tasks.ceph.mgr.x:Started 2023-01-24T16:58:58.539 INFO:tasks.ceph:Setting crush tunables to default 2023-01-24T16:58:58.540 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd crush tunables default 2023-01-24T16:58:58.775 INFO:teuthology.orchestra.run.smithi103.stderr:adjusted tunables profile to default 2023-01-24T16:58:58.786 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-01-24T16:58:58.786 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:58.787 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-01-24T16:58:58.800 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:58.801 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-01-24T16:58:58.810 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:58:58.810 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-01-24T16:58:58.868 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new d7b517d7-6f64-4813-8924-2a5c63538701 0 2023-01-24T16:58:58.872 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:58:58.860+0000 7f86bfaef000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-01-24T16:58:59.058 INFO:teuthology.orchestra.run.smithi103.stdout:0 2023-01-24T16:58:59.069 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new f14ded86-755d-468d-85eb-95f1f6fb298a 1 2023-01-24T16:58:59.265 INFO:teuthology.orchestra.run.smithi103.stdout:1 2023-01-24T16:58:59.282 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd new 2e034e0e-0171-4de2-80aa-2f7f6ef8c2c6 2 2023-01-24T16:58:59.476 INFO:teuthology.orchestra.run.smithi103.stdout:2 2023-01-24T16:58:59.477 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:58:59.468+0000 7f86bfaef000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-01-24T16:58:59.488 INFO:tasks.ceph.osd.0:Restarting daemon 2023-01-24T16:58:59.488 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-01-24T16:58:59.490 INFO:tasks.ceph.osd.0:Started 2023-01-24T16:58:59.490 INFO:tasks.ceph.osd.1:Restarting daemon 2023-01-24T16:58:59.491 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-01-24T16:58:59.493 INFO:tasks.ceph.osd.1:Started 2023-01-24T16:58:59.493 INFO:tasks.ceph.osd.2:Restarting daemon 2023-01-24T16:58:59.493 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-01-24T16:58:59.495 INFO:tasks.ceph.osd.2:Started 2023-01-24T16:58:59.495 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T16:58:59.560 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:58:59.548+0000 7f86bfaef000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-01-24T16:58:59.674 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:58:59.664+0000 7f86bfaef000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-01-24T16:58:59.675 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:58:59.676 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":5,"fsid":"bf3b1473-4375-4731-8219-58951a76fe3f","created":"2023-01-24T16:58:58.666618+0000","modified":"2023-01-24T16:58:59.466576+0000","last_up_change":"0.000000","last_in_change":"2023-01-24T16:58:59.466576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d7b517d7-6f64-4813-8924-2a5c63538701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f14ded86-755d-468d-85eb-95f1f6fb298a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2e034e0e-0171-4de2-80aa-2f7f6ef8c2c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T16:58:59.687 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-01-24T16:58:59.687 INFO:tasks.ceph:Waiting for OSDs to come up 2023-01-24T16:58:59.871 INFO:tasks.ceph.osd.0.smithi103.stderr:2023-01-24T16:58:59.860+0000 7f21f4dc43c0 -1 Falling back to public interface 2023-01-24T16:58:59.872 INFO:tasks.ceph.osd.2.smithi103.stderr:2023-01-24T16:58:59.860+0000 7fd9537fc3c0 -1 Falling back to public interface 2023-01-24T16:58:59.896 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:58:59.884+0000 7f86bfaef000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-01-24T16:58:59.944 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:58:59.932+0000 7f86bfaef000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-01-24T16:58:59.990 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T16:59:00.123 INFO:tasks.ceph.osd.1.smithi103.stderr:2023-01-24T16:59:00.112+0000 7f3fb5cbc3c0 -1 Falling back to public interface 2023-01-24T16:59:00.140 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:00.128+0000 7f86bfaef000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-01-24T16:59:00.160 INFO:teuthology.misc.health.smithi103.stdout: 2023-01-24T16:59:00.161 INFO:teuthology.misc.health.smithi103.stdout:{"epoch":5,"fsid":"bf3b1473-4375-4731-8219-58951a76fe3f","created":"2023-01-24T16:58:58.666618+0000","modified":"2023-01-24T16:58:59.466576+0000","last_up_change":"0.000000","last_in_change":"2023-01-24T16:58:59.466576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d7b517d7-6f64-4813-8924-2a5c63538701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f14ded86-755d-468d-85eb-95f1f6fb298a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2e034e0e-0171-4de2-80aa-2f7f6ef8c2c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T16:59:00.170 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-01-24T16:59:00.196 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:00.184+0000 7f86bfaef000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-01-24T16:59:00.257 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:00.244+0000 7f86bfaef000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-01-24T16:59:00.678 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:00.668+0000 7f86bfaef000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-01-24T16:59:00.800 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:00.788+0000 7f86bfaef000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-01-24T16:59:01.025 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:01.016+0000 7f86bfaef000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-01-24T16:59:01.284 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:01.272+0000 7f86bfaef000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-01-24T16:59:01.349 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:01.336+0000 7f86bfaef000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-01-24T16:59:01.401 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:01.388+0000 7f86bfaef000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-01-24T16:59:01.456 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:01.444+0000 7f86bfaef000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-01-24T16:59:01.623 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:01.612+0000 7f86bfaef000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-01-24T16:59:01.788 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:01.776+0000 7f86bfaef000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-01-24T16:59:01.852 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:01.840+0000 7f86bfaef000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-01-24T16:59:02.011 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:02.000+0000 7f86bfaef000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-01-24T16:59:02.076 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:02.064+0000 7f86bfaef000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-01-24T16:59:02.685 INFO:tasks.ceph.osd.2.smithi103.stderr:2023-01-24T16:59:02.672+0000 7fd9537fc3c0 -1 osd.2 0 log_to_monitors true 2023-01-24T16:59:02.771 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:02.760+0000 7f86bfaef000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-01-24T16:59:02.887 INFO:tasks.ceph.osd.0.smithi103.stderr:2023-01-24T16:59:02.876+0000 7f21f4dc43c0 -1 osd.0 0 log_to_monitors true 2023-01-24T16:59:03.162 INFO:tasks.ceph.osd.1.smithi103.stderr:2023-01-24T16:59:03.152+0000 7f3fb5cbc3c0 -1 osd.1 0 log_to_monitors true 2023-01-24T16:59:03.807 INFO:tasks.ceph.mgr.x.smithi103.stderr:2023-01-24T16:59:03.796+0000 7f86a8c30700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-01-24T16:59:04.578 INFO:tasks.ceph.osd.2.smithi103.stderr:2023-01-24T16:59:04.568+0000 7fd94ef9c700 -1 osd.2 0 waiting for initial osdmap 2023-01-24T16:59:04.585 INFO:tasks.ceph.osd.2.smithi103.stderr:2023-01-24T16:59:04.572+0000 7fd947d3f700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T16:59:05.799 INFO:tasks.ceph.osd.1.smithi103.stderr:2023-01-24T16:59:05.788+0000 7f3fb145c700 -1 osd.1 0 waiting for initial osdmap 2023-01-24T16:59:05.805 INFO:tasks.ceph.osd.1.smithi103.stderr:2023-01-24T16:59:05.796+0000 7f3faa1ff700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T16:59:05.885 INFO:tasks.ceph.osd.0.smithi103.stderr:2023-01-24T16:59:05.872+0000 7f21f0564700 -1 osd.0 0 waiting for initial osdmap 2023-01-24T16:59:05.892 INFO:tasks.ceph.osd.0.smithi103.stderr:2023-01-24T16:59:05.880+0000 7f21e9307700 -1 osd.0 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-24T16:59:06.474 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T16:59:06.750 INFO:teuthology.misc.health.smithi103.stdout: 2023-01-24T16:59:06.750 INFO:teuthology.misc.health.smithi103.stdout:{"epoch":9,"fsid":"bf3b1473-4375-4731-8219-58951a76fe3f","created":"2023-01-24T16:58:58.666618+0000","modified":"2023-01-24T16:59:05.823812+0000","last_up_change":"2023-01-24T16:59:05.823812+0000","last_in_change":"2023-01-24T16:58:59.466576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"d7b517d7-6f64-4813-8924-2a5c63538701","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f14ded86-755d-468d-85eb-95f1f6fb298a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":25682},{"type":"v1","addr":"172.21.15.103:6817","nonce":25682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":25682},{"type":"v1","addr":"172.21.15.103:6819","nonce":25682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":25682},{"type":"v1","addr":"172.21.15.103:6823","nonce":25682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":25682},{"type":"v1","addr":"172.21.15.103:6821","nonce":25682}]},"public_addr":"172.21.15.103:6817/25682","cluster_addr":"172.21.15.103:6819/25682","heartbeat_back_addr":"172.21.15.103:6823/25682","heartbeat_front_addr":"172.21.15.103:6821/25682","state":["exists","up"]},{"osd":2,"uuid":"2e034e0e-0171-4de2-80aa-2f7f6ef8c2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":25683},{"type":"v1","addr":"172.21.15.103:6807","nonce":25683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":25683},{"type":"v1","addr":"172.21.15.103:6811","nonce":25683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":25683},{"type":"v1","addr":"172.21.15.103:6815","nonce":25683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":25683},{"type":"v1","addr":"172.21.15.103:6813","nonce":25683}]},"public_addr":"172.21.15.103:6807/25683","cluster_addr":"172.21.15.103:6811/25683","heartbeat_back_addr":"172.21.15.103:6815/25683","heartbeat_front_addr":"172.21.15.103:6813/25683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:59:03.725013+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T16:59:06.763 DEBUG:teuthology.misc:2 of 3 OSDs are up 2023-01-24T16:59:13.066 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-24T16:59:13.341 INFO:teuthology.misc.health.smithi103.stdout: 2023-01-24T16:59:13.342 INFO:teuthology.misc.health.smithi103.stdout:{"epoch":12,"fsid":"bf3b1473-4375-4731-8219-58951a76fe3f","created":"2023-01-24T16:58:58.666618+0000","modified":"2023-01-24T16:59:09.635462+0000","last_up_change":"2023-01-24T16:59:06.881220+0000","last_in_change":"2023-01-24T16:58:59.466576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T16:59:06.853115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d7b517d7-6f64-4813-8924-2a5c63538701","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":25680},{"type":"v1","addr":"172.21.15.103:6801","nonce":25680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":25680},{"type":"v1","addr":"172.21.15.103:6803","nonce":25680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":25680},{"type":"v1","addr":"172.21.15.103:6810","nonce":25680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":25680},{"type":"v1","addr":"172.21.15.103:6806","nonce":25680}]},"public_addr":"172.21.15.103:6801/25680","cluster_addr":"172.21.15.103:6803/25680","heartbeat_back_addr":"172.21.15.103:6810/25680","heartbeat_front_addr":"172.21.15.103:6806/25680","state":["exists","up"]},{"osd":1,"uuid":"f14ded86-755d-468d-85eb-95f1f6fb298a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":25682},{"type":"v1","addr":"172.21.15.103:6817","nonce":25682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":25682},{"type":"v1","addr":"172.21.15.103:6819","nonce":25682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":25682},{"type":"v1","addr":"172.21.15.103:6823","nonce":25682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":25682},{"type":"v1","addr":"172.21.15.103:6821","nonce":25682}]},"public_addr":"172.21.15.103:6817/25682","cluster_addr":"172.21.15.103:6819/25682","heartbeat_back_addr":"172.21.15.103:6823/25682","heartbeat_front_addr":"172.21.15.103:6821/25682","state":["exists","up"]},{"osd":2,"uuid":"2e034e0e-0171-4de2-80aa-2f7f6ef8c2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":25683},{"type":"v1","addr":"172.21.15.103:6807","nonce":25683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":25683},{"type":"v1","addr":"172.21.15.103:6811","nonce":25683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":25683},{"type":"v1","addr":"172.21.15.103:6815","nonce":25683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":25683},{"type":"v1","addr":"172.21.15.103:6813","nonce":25683}]},"public_addr":"172.21.15.103:6807/25683","cluster_addr":"172.21.15.103:6811/25683","heartbeat_back_addr":"172.21.15.103:6815/25683","heartbeat_front_addr":"172.21.15.103:6813/25683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:59:03.885125+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:59:04.187806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:59:03.725013+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T16:59:13.353 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-01-24T16:59:13.354 INFO:tasks.ceph:Creating RBD pool 2023-01-24T16:59:13.354 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd pool create rbd 8 2023-01-24T16:59:14.163 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'rbd' created 2023-01-24T16:59:14.176 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-01-24T16:59:15.172 INFO:teuthology.orchestra.run.smithi103.stderr:enabled application 'rbd' on pool 'rbd' 2023-01-24T16:59:15.183 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-01-24T16:59:15.184 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-01-24T16:59:15.184 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-01-24T16:59:15.185 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-01-24T16:59:15.185 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-01-24T16:59:15.502 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:59:15.514 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":25496},{"type":"v1","addr":"172.21.15.103:6825","nonce":25496}]},"active_addr":"172.21.15.103:6825/25496","active_change":"2023-01-24T16:59:02.785762+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":1048885471}]},{"addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":693884366}]},{"addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":257084367}]},{"addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":3759306607}]}]}} 2023-01-24T16:59:15.518 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-01-24T16:59:15.519 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-01-24T16:59:15.519 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T16:59:15.807 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:59:15.809 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":14,"fsid":"bf3b1473-4375-4731-8219-58951a76fe3f","created":"2023-01-24T16:58:58.666618+0000","modified":"2023-01-24T16:59:15.145104+0000","last_up_change":"2023-01-24T16:59:06.881220+0000","last_in_change":"2023-01-24T16:58:59.466576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T16:59:06.853115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-24T16:59:13.672144+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d7b517d7-6f64-4813-8924-2a5c63538701","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":25680},{"type":"v1","addr":"172.21.15.103:6801","nonce":25680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":25680},{"type":"v1","addr":"172.21.15.103:6803","nonce":25680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":25680},{"type":"v1","addr":"172.21.15.103:6810","nonce":25680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":25680},{"type":"v1","addr":"172.21.15.103:6806","nonce":25680}]},"public_addr":"172.21.15.103:6801/25680","cluster_addr":"172.21.15.103:6803/25680","heartbeat_back_addr":"172.21.15.103:6810/25680","heartbeat_front_addr":"172.21.15.103:6806/25680","state":["exists","up"]},{"osd":1,"uuid":"f14ded86-755d-468d-85eb-95f1f6fb298a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":25682},{"type":"v1","addr":"172.21.15.103:6817","nonce":25682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":25682},{"type":"v1","addr":"172.21.15.103:6819","nonce":25682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":25682},{"type":"v1","addr":"172.21.15.103:6823","nonce":25682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":25682},{"type":"v1","addr":"172.21.15.103:6821","nonce":25682}]},"public_addr":"172.21.15.103:6817/25682","cluster_addr":"172.21.15.103:6819/25682","heartbeat_back_addr":"172.21.15.103:6823/25682","heartbeat_front_addr":"172.21.15.103:6821/25682","state":["exists","up"]},{"osd":2,"uuid":"2e034e0e-0171-4de2-80aa-2f7f6ef8c2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":25683},{"type":"v1","addr":"172.21.15.103:6807","nonce":25683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":25683},{"type":"v1","addr":"172.21.15.103:6811","nonce":25683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":25683},{"type":"v1","addr":"172.21.15.103:6815","nonce":25683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":25683},{"type":"v1","addr":"172.21.15.103:6813","nonce":25683}]},"public_addr":"172.21.15.103:6807/25683","cluster_addr":"172.21.15.103:6811/25683","heartbeat_back_addr":"172.21.15.103:6815/25683","heartbeat_front_addr":"172.21.15.103:6813/25683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:59:03.885125+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:59:04.187806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:59:03.725013+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T16:59:15.819 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-01-24T16:59:15.820 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T16:59:16.103 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:59:16.103 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":14,"fsid":"bf3b1473-4375-4731-8219-58951a76fe3f","created":"2023-01-24T16:58:58.666618+0000","modified":"2023-01-24T16:59:15.145104+0000","last_up_change":"2023-01-24T16:59:06.881220+0000","last_in_change":"2023-01-24T16:58:59.466576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T16:59:06.853115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-24T16:59:13.672144+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"d7b517d7-6f64-4813-8924-2a5c63538701","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":25680},{"type":"v1","addr":"172.21.15.103:6801","nonce":25680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":25680},{"type":"v1","addr":"172.21.15.103:6803","nonce":25680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":25680},{"type":"v1","addr":"172.21.15.103:6810","nonce":25680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":25680},{"type":"v1","addr":"172.21.15.103:6806","nonce":25680}]},"public_addr":"172.21.15.103:6801/25680","cluster_addr":"172.21.15.103:6803/25680","heartbeat_back_addr":"172.21.15.103:6810/25680","heartbeat_front_addr":"172.21.15.103:6806/25680","state":["exists","up"]},{"osd":1,"uuid":"f14ded86-755d-468d-85eb-95f1f6fb298a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":25682},{"type":"v1","addr":"172.21.15.103:6817","nonce":25682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":25682},{"type":"v1","addr":"172.21.15.103:6819","nonce":25682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":25682},{"type":"v1","addr":"172.21.15.103:6823","nonce":25682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":25682},{"type":"v1","addr":"172.21.15.103:6821","nonce":25682}]},"public_addr":"172.21.15.103:6817/25682","cluster_addr":"172.21.15.103:6819/25682","heartbeat_back_addr":"172.21.15.103:6823/25682","heartbeat_front_addr":"172.21.15.103:6821/25682","state":["exists","up"]},{"osd":2,"uuid":"2e034e0e-0171-4de2-80aa-2f7f6ef8c2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":25683},{"type":"v1","addr":"172.21.15.103:6807","nonce":25683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":25683},{"type":"v1","addr":"172.21.15.103:6811","nonce":25683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":25683},{"type":"v1","addr":"172.21.15.103:6815","nonce":25683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":25683},{"type":"v1","addr":"172.21.15.103:6813","nonce":25683}]},"public_addr":"172.21.15.103:6807/25683","cluster_addr":"172.21.15.103:6811/25683","heartbeat_back_addr":"172.21.15.103:6815/25683","heartbeat_front_addr":"172.21.15.103:6813/25683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:59:03.885125+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:59:04.187806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:59:03.725013+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T16:59:16.117 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-01-24T16:59:16.117 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-01-24T16:59:16.118 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-01-24T16:59:16.243 INFO:teuthology.orchestra.run.smithi103.stdout:42949672965 2023-01-24T16:59:16.244 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-24T16:59:16.258 INFO:teuthology.orchestra.run.smithi103.stdout:34359738372 2023-01-24T16:59:16.258 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-24T16:59:16.261 INFO:teuthology.orchestra.run.smithi103.stdout:38654705669 2023-01-24T16:59:16.261 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-24T16:59:16.532 INFO:teuthology.orchestra.run.smithi103.stdout:38654705668 2023-01-24T16:59:16.534 INFO:teuthology.orchestra.run.smithi103.stdout:42949672964 2023-01-24T16:59:16.536 INFO:teuthology.orchestra.run.smithi103.stdout:34359738370 2023-01-24T16:59:16.546 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705668 for osd.1 2023-01-24T16:59:16.548 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 42949672964 for osd.0 2023-01-24T16:59:16.550 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738370 for osd.2 2023-01-24T16:59:17.547 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-24T16:59:17.548 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-24T16:59:17.551 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-24T16:59:17.830 INFO:teuthology.orchestra.run.smithi103.stdout:38654705670 2023-01-24T16:59:17.834 INFO:teuthology.orchestra.run.smithi103.stdout:42949672965 2023-01-24T16:59:17.848 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 42949672965 for osd.0 2023-01-24T16:59:17.849 DEBUG:teuthology.parallel:result is None 2023-01-24T16:59:17.849 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705669 got 38654705670 for osd.1 2023-01-24T16:59:17.849 DEBUG:teuthology.parallel:result is None 2023-01-24T16:59:17.871 INFO:teuthology.orchestra.run.smithi103.stdout:34359738372 2023-01-24T16:59:17.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.2 2023-01-24T16:59:17.884 DEBUG:teuthology.parallel:result is None 2023-01-24T16:59:17.884 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-01-24T16:59:17.885 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T16:59:18.163 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:59:18.166 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2023-01-24T16:59:18.175 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-01-24T16:59:16.797561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24968,"kb_used_data":1520,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547832,"statfs":{"total":322122547200,"available":322096979968,"internally_reserved":0,"allocated":1556480,"data_stored":1280333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001031"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.173563+0000","last_change":"2023-01-24T16:59:15.168780+0000","last_active":"2023-01-24T16:59:15.173563+0000","last_peered":"2023-01-24T16:59:15.173563+0000","last_clean":"2023-01-24T16:59:15.173563+0000","last_became_active":"2023-01-24T16:59:15.168484+0000","last_became_peered":"2023-01-24T16:59:15.168484+0000","last_unstale":"2023-01-24T16:59:15.173563+0000","last_undegraded":"2023-01-24T16:59:15.173563+0000","last_fullsized":"2023-01-24T16:59:15.173563+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_clean_scrub_stamp":"2023-01-24T16:59:14.137617+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T01:13:42.640899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.173583+0000","last_change":"2023-01-24T16:59:15.169982+0000","last_active":"2023-01-24T16:59:15.173583+0000","last_peered":"2023-01-24T16:59:15.173583+0000","last_clean":"2023-01-24T16:59:15.173583+0000","last_became_active":"2023-01-24T16:59:15.169710+0000","last_became_peered":"2023-01-24T16:59:15.169710+0000","last_unstale":"2023-01-24T16:59:15.173583+0000","last_undegraded":"2023-01-24T16:59:15.173583+0000","last_fullsized":"2023-01-24T16:59:15.173583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_clean_scrub_stamp":"2023-01-24T16:59:14.137617+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T19:17:35.088291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.173574+0000","last_change":"2023-01-24T16:59:15.169145+0000","last_active":"2023-01-24T16:59:15.173574+0000","last_peered":"2023-01-24T16:59:15.173574+0000","last_clean":"2023-01-24T16:59:15.173574+0000","last_became_active":"2023-01-24T16:59:15.168904+0000","last_became_peered":"2023-01-24T16:59:15.168904+0000","last_unstale":"2023-01-24T16:59:15.173574+0000","last_undegraded":"2023-01-24T16:59:15.173574+0000","last_fullsized":"2023-01-24T16:59:15.173574+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_clean_scrub_stamp":"2023-01-24T16:59:14.137617+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T17:01:34.175320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.173596+0000","last_change":"2023-01-24T16:59:15.168940+0000","last_active":"2023-01-24T16:59:15.173596+0000","last_peered":"2023-01-24T16:59:15.173596+0000","last_clean":"2023-01-24T16:59:15.173596+0000","last_became_active":"2023-01-24T16:59:15.168630+0000","last_became_peered":"2023-01-24T16:59:15.168630+0000","last_unstale":"2023-01-24T16:59:15.173596+0000","last_undegraded":"2023-01-24T16:59:15.173596+0000","last_fullsized":"2023-01-24T16:59:15.173596+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_clean_scrub_stamp":"2023-01-24T16:59:14.137617+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T18:49:03.502281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.168393+0000","last_change":"2023-01-24T16:59:15.166556+0000","last_active":"2023-01-24T16:59:15.168393+0000","last_peered":"2023-01-24T16:59:15.168393+0000","last_clean":"2023-01-24T16:59:15.168393+0000","last_became_active":"2023-01-24T16:59:15.166344+0000","last_became_peered":"2023-01-24T16:59:15.166344+0000","last_unstale":"2023-01-24T16:59:15.168393+0000","last_undegraded":"2023-01-24T16:59:15.168393+0000","last_fullsized":"2023-01-24T16:59:15.168393+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_clean_scrub_stamp":"2023-01-24T16:59:14.137617+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T01:29:58.362664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.165837+0000","last_change":"2023-01-24T16:59:15.165837+0000","last_active":"2023-01-24T16:59:15.165837+0000","last_peered":"2023-01-24T16:59:15.165837+0000","last_clean":"2023-01-24T16:59:15.165837+0000","last_became_active":"2023-01-24T16:59:15.165510+0000","last_became_peered":"2023-01-24T16:59:15.165510+0000","last_unstale":"2023-01-24T16:59:15.165837+0000","last_undegraded":"2023-01-24T16:59:15.165837+0000","last_fullsized":"2023-01-24T16:59:15.165837+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_clean_scrub_stamp":"2023-01-24T16:59:14.137617+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T19:07:43.194559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.166776+0000","last_change":"2023-01-24T16:59:15.166776+0000","last_active":"2023-01-24T16:59:15.166776+0000","last_peered":"2023-01-24T16:59:15.166776+0000","last_clean":"2023-01-24T16:59:15.166776+0000","last_became_active":"2023-01-24T16:59:15.166358+0000","last_became_peered":"2023-01-24T16:59:15.166358+0000","last_unstale":"2023-01-24T16:59:15.166776+0000","last_undegraded":"2023-01-24T16:59:15.166776+0000","last_fullsized":"2023-01-24T16:59:15.166776+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_clean_scrub_stamp":"2023-01-24T16:59:14.137617+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T23:23:22.228305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.170824+0000","last_change":"2023-01-24T16:59:15.166026+0000","last_active":"2023-01-24T16:59:15.170824+0000","last_peered":"2023-01-24T16:59:15.170824+0000","last_clean":"2023-01-24T16:59:15.170824+0000","last_became_active":"2023-01-24T16:59:15.165672+0000","last_became_peered":"2023-01-24T16:59:15.165672+0000","last_unstale":"2023-01-24T16:59:15.170824+0000","last_undegraded":"2023-01-24T16:59:15.170824+0000","last_fullsized":"2023-01-24T16:59:15.170824+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_clean_scrub_stamp":"2023-01-24T16:59:14.137617+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T20:28:52.136106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.155543+0000","last_change":"2023-01-24T16:59:08.798542+0000","last_active":"2023-01-24T16:59:15.155543+0000","last_peered":"2023-01-24T16:59:15.155543+0000","last_clean":"2023-01-24T16:59:15.155543+0000","last_became_active":"2023-01-24T16:59:08.796015+0000","last_became_peered":"2023-01-24T16:59:08.796015+0000","last_unstale":"2023-01-24T16:59:15.155543+0000","last_undegraded":"2023-01-24T16:59:15.155543+0000","last_fullsized":"2023-01-24T16:59:15.155543+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:06.881220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:06.881220+0000","last_clean_scrub_stamp":"2023-01-24T16:59:06.881220+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T01:30:52.251681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672965,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8644,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848956,"statfs":{"total":107374182400,"available":107365330944,"internally_reserved":0,"allocated":716800,"data_stored":623567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]}]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8644,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848956,"statfs":{"total":107374182400,"available":107365330944,"internally_reserved":0,"allocated":716800,"data_stored":623567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7680,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849920,"statfs":{"total":107374182400,"available":107366318080,"internally_reserved":0,"allocated":122880,"data_stored":33199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-24T16:59:18.176 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-24T16:59:18.458 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:59:18.461 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2023-01-24T16:59:18.477 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-01-24T16:59:16.797561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24968,"kb_used_data":1520,"kb_used_omap":0,"kb_used_meta":23424,"kb_avail":314547832,"statfs":{"total":322122547200,"available":322096979968,"internally_reserved":0,"allocated":1556480,"data_stored":1280333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23986176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001031"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.173563+0000","last_change":"2023-01-24T16:59:15.168780+0000","last_active":"2023-01-24T16:59:15.173563+0000","last_peered":"2023-01-24T16:59:15.173563+0000","last_clean":"2023-01-24T16:59:15.173563+0000","last_became_active":"2023-01-24T16:59:15.168484+0000","last_became_peered":"2023-01-24T16:59:15.168484+0000","last_unstale":"2023-01-24T16:59:15.173563+0000","last_undegraded":"2023-01-24T16:59:15.173563+0000","last_fullsized":"2023-01-24T16:59:15.173563+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_clean_scrub_stamp":"2023-01-24T16:59:14.137617+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T01:13:42.640899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.173583+0000","last_change":"2023-01-24T16:59:15.169982+0000","last_active":"2023-01-24T16:59:15.173583+0000","last_peered":"2023-01-24T16:59:15.173583+0000","last_clean":"2023-01-24T16:59:15.173583+0000","last_became_active":"2023-01-24T16:59:15.169710+0000","last_became_peered":"2023-01-24T16:59:15.169710+0000","last_unstale":"2023-01-24T16:59:15.173583+0000","last_undegraded":"2023-01-24T16:59:15.173583+0000","last_fullsized":"2023-01-24T16:59:15.173583+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_clean_scrub_stamp":"2023-01-24T16:59:14.137617+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T19:17:35.088291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.173574+0000","last_change":"2023-01-24T16:59:15.169145+0000","last_active":"2023-01-24T16:59:15.173574+0000","last_peered":"2023-01-24T16:59:15.173574+0000","last_clean":"2023-01-24T16:59:15.173574+0000","last_became_active":"2023-01-24T16:59:15.168904+0000","last_became_peered":"2023-01-24T16:59:15.168904+0000","last_unstale":"2023-01-24T16:59:15.173574+0000","last_undegraded":"2023-01-24T16:59:15.173574+0000","last_fullsized":"2023-01-24T16:59:15.173574+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_clean_scrub_stamp":"2023-01-24T16:59:14.137617+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T17:01:34.175320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.173596+0000","last_change":"2023-01-24T16:59:15.168940+0000","last_active":"2023-01-24T16:59:15.173596+0000","last_peered":"2023-01-24T16:59:15.173596+0000","last_clean":"2023-01-24T16:59:15.173596+0000","last_became_active":"2023-01-24T16:59:15.168630+0000","last_became_peered":"2023-01-24T16:59:15.168630+0000","last_unstale":"2023-01-24T16:59:15.173596+0000","last_undegraded":"2023-01-24T16:59:15.173596+0000","last_fullsized":"2023-01-24T16:59:15.173596+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_clean_scrub_stamp":"2023-01-24T16:59:14.137617+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T18:49:03.502281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.168393+0000","last_change":"2023-01-24T16:59:15.166556+0000","last_active":"2023-01-24T16:59:15.168393+0000","last_peered":"2023-01-24T16:59:15.168393+0000","last_clean":"2023-01-24T16:59:15.168393+0000","last_became_active":"2023-01-24T16:59:15.166344+0000","last_became_peered":"2023-01-24T16:59:15.166344+0000","last_unstale":"2023-01-24T16:59:15.168393+0000","last_undegraded":"2023-01-24T16:59:15.168393+0000","last_fullsized":"2023-01-24T16:59:15.168393+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_clean_scrub_stamp":"2023-01-24T16:59:14.137617+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T01:29:58.362664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.165837+0000","last_change":"2023-01-24T16:59:15.165837+0000","last_active":"2023-01-24T16:59:15.165837+0000","last_peered":"2023-01-24T16:59:15.165837+0000","last_clean":"2023-01-24T16:59:15.165837+0000","last_became_active":"2023-01-24T16:59:15.165510+0000","last_became_peered":"2023-01-24T16:59:15.165510+0000","last_unstale":"2023-01-24T16:59:15.165837+0000","last_undegraded":"2023-01-24T16:59:15.165837+0000","last_fullsized":"2023-01-24T16:59:15.165837+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_clean_scrub_stamp":"2023-01-24T16:59:14.137617+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T19:07:43.194559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.166776+0000","last_change":"2023-01-24T16:59:15.166776+0000","last_active":"2023-01-24T16:59:15.166776+0000","last_peered":"2023-01-24T16:59:15.166776+0000","last_clean":"2023-01-24T16:59:15.166776+0000","last_became_active":"2023-01-24T16:59:15.166358+0000","last_became_peered":"2023-01-24T16:59:15.166358+0000","last_unstale":"2023-01-24T16:59:15.166776+0000","last_undegraded":"2023-01-24T16:59:15.166776+0000","last_fullsized":"2023-01-24T16:59:15.166776+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_clean_scrub_stamp":"2023-01-24T16:59:14.137617+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T23:23:22.228305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.170824+0000","last_change":"2023-01-24T16:59:15.166026+0000","last_active":"2023-01-24T16:59:15.170824+0000","last_peered":"2023-01-24T16:59:15.170824+0000","last_clean":"2023-01-24T16:59:15.170824+0000","last_became_active":"2023-01-24T16:59:15.165672+0000","last_became_peered":"2023-01-24T16:59:15.165672+0000","last_unstale":"2023-01-24T16:59:15.170824+0000","last_undegraded":"2023-01-24T16:59:15.170824+0000","last_fullsized":"2023-01-24T16:59:15.170824+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:14.137617+0000","last_clean_scrub_stamp":"2023-01-24T16:59:14.137617+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-25T20:28:52.136106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-01-24T16:59:15.155543+0000","last_change":"2023-01-24T16:59:08.798542+0000","last_active":"2023-01-24T16:59:15.155543+0000","last_peered":"2023-01-24T16:59:15.155543+0000","last_clean":"2023-01-24T16:59:15.155543+0000","last_became_active":"2023-01-24T16:59:08.796015+0000","last_became_peered":"2023-01-24T16:59:08.796015+0000","last_unstale":"2023-01-24T16:59:15.155543+0000","last_undegraded":"2023-01-24T16:59:15.155543+0000","last_fullsized":"2023-01-24T16:59:15.155543+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-24T16:59:06.881220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-24T16:59:06.881220+0000","last_clean_scrub_stamp":"2023-01-24T16:59:06.881220+0000","objects_scrubbed":0,"log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T01:30:52.251681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":10,"seq":42949672965,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8644,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848956,"statfs":{"total":107374182400,"available":107365330944,"internally_reserved":0,"allocated":716800,"data_stored":623567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]}]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8644,"kb_used_data":700,"kb_used_omap":0,"kb_used_meta":7936,"kb_avail":104848956,"statfs":{"total":107374182400,"available":107365330944,"internally_reserved":0,"allocated":716800,"data_stored":623567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8126464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7680,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":7552,"kb_avail":104849920,"statfs":{"total":107374182400,"available":107366318080,"internally_reserved":0,"allocated":122880,"data_stored":33199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7733248},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-24T16:59:18.479 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-01-24T16:59:18.479 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-01-24T16:59:18.479 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-01-24T16:59:18.480 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-01-24T16:59:18.791 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T16:59:18.792 INFO:teuthology.orchestra.run.smithi103.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-01-24T16:59:18.804 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-01-24T16:59:18.805 INFO:teuthology.run_tasks:Running task openssl_keys... 2023-01-24T16:59:18.820 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p /home/ubuntu/cephtest/ca 2023-01-24T16:59:18.826 DEBUG:teuthology.orchestra.run.smithi103:> openssl req -x509 -nodes -newkey rsa:4096 -keyout /home/ubuntu/cephtest/ca/kmiproot.key -days 365 -out /home/ubuntu/cephtest/ca/kmiproot.crt -subj /CN=kmiproot 2023-01-24T16:59:18.894 INFO:teuthology.orchestra.run.smithi103.stderr:Generating a RSA private key 2023-01-24T16:59:19.422 INFO:teuthology.orchestra.run.smithi103.stderr:.................................................................................................................................................................++++ 2023-01-24T16:59:19.528 INFO:teuthology.orchestra.run.smithi103.stderr:................................++++ 2023-01-24T16:59:19.528 INFO:teuthology.orchestra.run.smithi103.stderr:writing new private key to '/home/ubuntu/cephtest/ca/kmiproot.key' 2023-01-24T16:59:19.529 INFO:teuthology.orchestra.run.smithi103.stderr:----- 2023-01-24T16:59:19.534 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p /home/ubuntu/cephtest/ca 2023-01-24T16:59:19.541 DEBUG:teuthology.orchestra.run.smithi103:> openssl req -nodes -newkey rsa:2048 -keyout /home/ubuntu/cephtest/ca/kmip-client.key -out /home/ubuntu/cephtest/ca/kmip-client.csr -subj /CN=rgw-client 2023-01-24T16:59:19.551 INFO:teuthology.orchestra.run.smithi103.stderr:Generating a RSA private key 2023-01-24T16:59:19.575 INFO:teuthology.orchestra.run.smithi103.stderr:...................+++++ 2023-01-24T16:59:19.635 INFO:teuthology.orchestra.run.smithi103.stderr:...........................................................+++++ 2023-01-24T16:59:19.635 INFO:teuthology.orchestra.run.smithi103.stderr:writing new private key to '/home/ubuntu/cephtest/ca/kmip-client.key' 2023-01-24T16:59:19.635 INFO:teuthology.orchestra.run.smithi103.stderr:----- 2023-01-24T16:59:19.637 DEBUG:teuthology.orchestra.run.smithi103:> openssl x509 -req -in /home/ubuntu/cephtest/ca/kmip-client.csr -CA /home/ubuntu/cephtest/ca/kmiproot.crt -CAkey /home/ubuntu/cephtest/ca/kmiproot.key -CAcreateserial -out /home/ubuntu/cephtest/ca/kmip-client.crt -days 365 -sha256 2023-01-24T16:59:19.643 INFO:teuthology.orchestra.run.smithi103.stderr:Signature ok 2023-01-24T16:59:19.643 INFO:teuthology.orchestra.run.smithi103.stderr:subject=CN = rgw-client 2023-01-24T16:59:19.643 INFO:teuthology.orchestra.run.smithi103.stderr:Getting CA Private Key 2023-01-24T16:59:19.650 DEBUG:teuthology.orchestra.run.smithi103:> rm /home/ubuntu/cephtest/ca/kmip-client.csr /home/ubuntu/cephtest/ca/kmiproot.srl 2023-01-24T16:59:19.703 DEBUG:teuthology.orchestra.run.smithi103:> openssl verify -CAfile /home/ubuntu/cephtest/ca/kmiproot.crt /home/ubuntu/cephtest/ca/kmip-client.crt 2023-01-24T16:59:19.713 INFO:teuthology.orchestra.run.smithi103.stdout:/home/ubuntu/cephtest/ca/kmip-client.crt: OK 2023-01-24T16:59:19.713 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p /home/ubuntu/cephtest/ca 2023-01-24T16:59:19.718 DEBUG:teuthology.orchestra.run.smithi103:> openssl req -nodes -newkey rsa:2048 -keyout /home/ubuntu/cephtest/ca/kmip-server.key -out /home/ubuntu/cephtest/ca/kmip-server.csr -subj /CN=smithi103.front.sepia.ceph.com 2023-01-24T16:59:19.728 INFO:teuthology.orchestra.run.smithi103.stderr:Generating a RSA private key 2023-01-24T16:59:19.798 INFO:teuthology.orchestra.run.smithi103.stderr:....................................................................+++++ 2023-01-24T16:59:20.033 INFO:teuthology.orchestra.run.smithi103.stderr:...................................................................................................................................................................................................................................................+++++ 2023-01-24T16:59:20.033 INFO:teuthology.orchestra.run.smithi103.stderr:writing new private key to '/home/ubuntu/cephtest/ca/kmip-server.key' 2023-01-24T16:59:20.033 INFO:teuthology.orchestra.run.smithi103.stderr:----- 2023-01-24T16:59:20.035 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T16:59:20.035 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/ca/kmip-server.ext 2023-01-24T16:59:20.042 DEBUG:teuthology.orchestra.run.smithi103:> openssl x509 -req -in /home/ubuntu/cephtest/ca/kmip-server.csr -CA /home/ubuntu/cephtest/ca/kmiproot.crt -CAkey /home/ubuntu/cephtest/ca/kmiproot.key -CAcreateserial -out /home/ubuntu/cephtest/ca/kmip-server.crt -days 365 -sha256 -extfile /home/ubuntu/cephtest/ca/kmip-server.ext 2023-01-24T16:59:20.095 INFO:teuthology.orchestra.run.smithi103.stderr:Signature ok 2023-01-24T16:59:20.095 INFO:teuthology.orchestra.run.smithi103.stderr:subject=CN = smithi103.front.sepia.ceph.com 2023-01-24T16:59:20.095 INFO:teuthology.orchestra.run.smithi103.stderr:Getting CA Private Key 2023-01-24T16:59:20.102 DEBUG:teuthology.orchestra.run.smithi103:> rm /home/ubuntu/cephtest/ca/kmip-server.csr /home/ubuntu/cephtest/ca/kmiproot.srl /home/ubuntu/cephtest/ca/kmip-server.ext 2023-01-24T16:59:20.109 DEBUG:teuthology.orchestra.run.smithi103:> openssl verify -CAfile /home/ubuntu/cephtest/ca/kmiproot.crt /home/ubuntu/cephtest/ca/kmip-server.crt 2023-01-24T16:59:20.116 INFO:teuthology.orchestra.run.smithi103.stdout:/home/ubuntu/cephtest/ca/kmip-server.crt: OK 2023-01-24T16:59:20.117 INFO:teuthology.run_tasks:Running task exec... 2023-01-24T16:59:20.127 INFO:teuthology.task.exec:Executing custom commands... 2023-01-24T16:59:20.127 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi103.front.sepia.ceph.com 2023-01-24T16:59:20.128 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'chmod 644 /home/ubuntu/cephtest/ca/kmip-client.key' 2023-01-24T16:59:20.173 INFO:teuthology.run_tasks:Running task pykmip... 2023-01-24T16:59:20.209 DEBUG:tasks.pykmip:PyKMIP config is {'client.0': {'clientca': 'kmiproot', 'clientcert': 'kmip-client', 'secrets': [{'name': 'pykmip-my-key-1'}, {'name': 'pykmip-my-key-2'}], 'servercert': 'kmip-server'}} 2023-01-24T16:59:20.209 INFO:tasks.pykmip:Downloading pykmip... 2023-01-24T16:59:20.210 INFO:tasks.pykmip:Using branch 'master' for pykmip 2023-01-24T16:59:20.210 INFO:tasks.pykmip:sha1=None 2023-01-24T16:59:20.210 DEBUG:teuthology.orchestra.run.smithi103:> git clone -b master https://github.com/OpenKMIP/PyKMIP /home/ubuntu/cephtest/pykmip 2023-01-24T16:59:20.217 INFO:teuthology.orchestra.run.smithi103.stderr:Cloning into '/home/ubuntu/cephtest/pykmip'... 2023-01-24T16:59:21.610 INFO:tasks.pykmip:Setting up virtualenv for pykmip... 2023-01-24T16:59:21.611 DEBUG:teuthology.orchestra.run.smithi103:> cd /home/ubuntu/cephtest/pykmip && python3 -m venv .pykmipenv 2023-01-24T16:59:23.209 DEBUG:teuthology.orchestra.run.smithi103:> cd /home/ubuntu/cephtest/pykmip && . .pykmipenv/bin/activate && pip install --upgrade pip 2023-01-24T16:59:24.378 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting pip 2023-01-24T16:59:24.540 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading pip-22.3.1-py3-none-any.whl (2.1 MB) 2023-01-24T16:59:25.070 INFO:teuthology.orchestra.run.smithi103.stdout:Installing collected packages: pip 2023-01-24T16:59:25.070 INFO:teuthology.orchestra.run.smithi103.stdout: Attempting uninstall: pip 2023-01-24T16:59:25.070 INFO:teuthology.orchestra.run.smithi103.stdout: Found existing installation: pip 20.0.2 2023-01-24T16:59:25.132 INFO:teuthology.orchestra.run.smithi103.stdout: Uninstalling pip-20.0.2: 2023-01-24T16:59:25.135 INFO:teuthology.orchestra.run.smithi103.stdout: Successfully uninstalled pip-20.0.2 2023-01-24T16:59:26.041 INFO:teuthology.orchestra.run.smithi103.stdout:Successfully installed pip-22.3.1 2023-01-24T16:59:26.085 DEBUG:teuthology.orchestra.run.smithi103:> cd /home/ubuntu/cephtest/pykmip && . .pykmipenv/bin/activate && pip install pytz -e /home/ubuntu/cephtest/pykmip 2023-01-24T16:59:26.498 INFO:teuthology.orchestra.run.smithi103.stdout:Obtaining file:///home/ubuntu/cephtest/pykmip 2023-01-24T16:59:26.499 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing metadata (setup.py): started 2023-01-24T16:59:26.741 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing metadata (setup.py): finished with status 'done' 2023-01-24T16:59:26.955 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting pytz 2023-01-24T16:59:27.032 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading pytz-2022.7.1-py2.py3-none-any.whl (499 kB) 2023-01-24T16:59:27.127 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 499.4/499.4 kB 5.6 MB/s eta 0:00:00 2023-01-24T16:59:27.459 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting cryptography 2023-01-24T16:59:27.479 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading cryptography-39.0.0-cp36-abi3-manylinux_2_28_x86_64.whl (4.2 MB) 2023-01-24T16:59:27.603 INFO:teuthology.orchestra.run.smithi103.stdout: ---------------------------------------- 4.2/4.2 MB 35.2 MB/s eta 0:00:00 2023-01-24T16:59:27.675 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting enum-compat 2023-01-24T16:59:27.697 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading enum_compat-0.0.3-py3-none-any.whl (1.3 kB) 2023-01-24T16:59:27.821 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting requests 2023-01-24T16:59:27.842 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading requests-2.28.2-py3-none-any.whl (62 kB) 2023-01-24T16:59:27.878 INFO:teuthology.orchestra.run.smithi103.stdout: ---------------------------------------- 62.8/62.8 kB 1.7 MB/s eta 0:00:00 2023-01-24T16:59:27.945 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting six 2023-01-24T16:59:27.962 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-01-24T16:59:28.402 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting sqlalchemy 2023-01-24T16:59:28.421 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading SQLAlchemy-1.4.46-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB) 2023-01-24T16:59:28.488 INFO:teuthology.orchestra.run.smithi103.stdout: ---------------------------------------- 1.6/1.6 MB 25.6 MB/s eta 0:00:00 2023-01-24T16:59:28.735 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting cffi>=1.12 2023-01-24T16:59:28.753 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading cffi-1.15.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (442 kB) 2023-01-24T16:59:28.788 INFO:teuthology.orchestra.run.smithi103.stdout: -------------------------------------- 442.7/442.7 kB 13.6 MB/s eta 0:00:00 2023-01-24T16:59:28.913 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting charset-normalizer<4,>=2 2023-01-24T16:59:28.954 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading charset_normalizer-3.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (195 kB) 2023-01-24T16:59:28.990 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 195.4/195.4 kB 5.8 MB/s eta 0:00:00 2023-01-24T16:59:29.077 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting urllib3<1.27,>=1.21.1 2023-01-24T16:59:29.096 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading urllib3-1.26.14-py2.py3-none-any.whl (140 kB) 2023-01-24T16:59:29.133 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 140.6/140.6 kB 3.8 MB/s eta 0:00:00 2023-01-24T16:59:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting idna<4,>=2.5 2023-01-24T16:59:29.222 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading idna-3.4-py3-none-any.whl (61 kB) 2023-01-24T16:59:29.251 INFO:teuthology.orchestra.run.smithi103.stdout: ---------------------------------------- 61.5/61.5 kB 2.1 MB/s eta 0:00:00 2023-01-24T16:59:29.328 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting certifi>=2017.4.17 2023-01-24T16:59:29.348 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading certifi-2022.12.7-py3-none-any.whl (155 kB) 2023-01-24T16:59:29.377 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 155.3/155.3 kB 6.0 MB/s eta 0:00:00 2023-01-24T16:59:29.700 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting greenlet!=0.4.17 2023-01-24T16:59:29.718 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading greenlet-2.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (544 kB) 2023-01-24T16:59:29.760 INFO:teuthology.orchestra.run.smithi103.stdout: -------------------------------------- 544.0/544.0 kB 13.9 MB/s eta 0:00:00 2023-01-24T16:59:29.831 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting pycparser 2023-01-24T16:59:29.850 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB) 2023-01-24T16:59:29.878 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 118.7/118.7 kB 4.5 MB/s eta 0:00:00 2023-01-24T16:59:30.036 INFO:teuthology.orchestra.run.smithi103.stdout:Installing collected packages: pytz, enum-compat, charset-normalizer, urllib3, six, pycparser, idna, greenlet, certifi, sqlalchemy, requests, cffi, cryptography, PyKMIP 2023-01-24T16:59:32.091 INFO:teuthology.orchestra.run.smithi103.stdout: Running setup.py develop for PyKMIP 2023-01-24T16:59:32.533 INFO:teuthology.orchestra.run.smithi103.stdout:Successfully installed PyKMIP-0.11.0.dev1 certifi-2022.12.7 cffi-1.15.1 charset-normalizer-3.0.1 cryptography-39.0.0 enum-compat-0.0.3 greenlet-2.0.1 idna-3.4 pycparser-2.21 pytz-2022.7.1 requests-2.28.2 six-1.16.0 sqlalchemy-1.4.46 urllib3-1.26.14 2023-01-24T16:59:32.858 INFO:tasks.pykmip:Installing system dependenies for PyKMIP... 2023-01-24T16:59:32.870 DEBUG:teuthology.orchestra.run.smithi103:> cd /home/ubuntu/cephtest/pykmip && . .pykmipenv/bin/activate && pip install bindep 2023-01-24T16:59:33.362 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting bindep 2023-01-24T16:59:33.436 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading bindep-2.11.0-py2.py3-none-any.whl (34 kB) 2023-01-24T16:59:33.552 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting packaging 2023-01-24T16:59:33.570 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading packaging-23.0-py3-none-any.whl (42 kB) 2023-01-24T16:59:33.607 INFO:teuthology.orchestra.run.smithi103.stdout: ---------------------------------------- 42.7/42.7 kB 1.0 MB/s eta 0:00:00 2023-01-24T16:59:33.666 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting Parsley 2023-01-24T16:59:33.684 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading Parsley-1.3-py2.py3-none-any.whl (88 kB) 2023-01-24T16:59:33.725 INFO:teuthology.orchestra.run.smithi103.stdout: ---------------------------------------- 88.9/88.9 kB 2.2 MB/s eta 0:00:00 2023-01-24T16:59:33.791 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting distro>=1.7.0 2023-01-24T16:59:33.808 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading distro-1.8.0-py3-none-any.whl (20 kB) 2023-01-24T16:59:33.923 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting pbr>=2.0.0 2023-01-24T16:59:33.940 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading pbr-5.11.1-py2.py3-none-any.whl (112 kB) 2023-01-24T16:59:33.969 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 112.7/112.7 kB 4.2 MB/s eta 0:00:00 2023-01-24T16:59:34.093 INFO:teuthology.orchestra.run.smithi103.stdout:Installing collected packages: Parsley, pbr, packaging, distro, bindep 2023-01-24T16:59:34.581 INFO:teuthology.orchestra.run.smithi103.stdout:Successfully installed Parsley-1.3 bindep-2.11.0 distro-1.8.0 packaging-23.0 pbr-5.11.1 2023-01-24T16:59:34.655 DEBUG:teuthology.orchestra.run.smithi103:> cd /home/ubuntu/cephtest/pykmip && . .pykmipenv/bin/activate && bindep --brief --file /tmp/bindep-f3cmgqbr.txt 2023-01-24T16:59:35.235 INFO:teuthology.orchestra.run.smithi103.stdout:libffi-dev 2023-01-24T16:59:35.236 INFO:teuthology.orchestra.run.smithi103.stdout:libsqlite3-dev 2023-01-24T16:59:35.236 INFO:teuthology.orchestra.run.smithi103.stdout:libssl-dev 2023-01-24T16:59:35.236 INFO:teuthology.orchestra.run.smithi103.stdout:python-dev 2023-01-24T16:59:35.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T16:59:35.400 INFO:teuthology.packaging:Installing package libffi-dev on ubuntu@smithi103.front.sepia.ceph.com 2023-01-24T16:59:35.401 DEBUG:teuthology.orchestra.run.smithi103:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install libffi-dev 2023-01-24T16:59:35.468 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2023-01-24T16:59:35.656 INFO:teuthology.orchestra.run.smithi103.stdout:Building dependency tree... 2023-01-24T16:59:35.657 INFO:teuthology.orchestra.run.smithi103.stdout:Reading state information... 2023-01-24T16:59:35.823 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages were automatically installed and are no longer required: 2023-01-24T16:59:35.824 INFO:teuthology.orchestra.run.smithi103.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-01-24T16:59:35.824 INFO:teuthology.orchestra.run.smithi103.stdout:Use 'sudo apt autoremove' to remove them. 2023-01-24T16:59:35.855 INFO:teuthology.orchestra.run.smithi103.stdout:The following NEW packages will be installed: 2023-01-24T16:59:35.856 INFO:teuthology.orchestra.run.smithi103.stdout: libffi-dev 2023-01-24T16:59:36.036 INFO:teuthology.orchestra.run.smithi103.stdout:0 upgraded, 1 newly installed, 0 to remove and 120 not upgraded. 2023-01-24T16:59:36.037 INFO:teuthology.orchestra.run.smithi103.stdout:Need to get 57.0 kB of archives. 2023-01-24T16:59:36.037 INFO:teuthology.orchestra.run.smithi103.stdout:After this operation, 312 kB of additional disk space will be used. 2023-01-24T16:59:36.037 INFO:teuthology.orchestra.run.smithi103.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libffi-dev amd64 3.3-4 [57.0 kB] 2023-01-24T16:59:36.569 INFO:teuthology.orchestra.run.smithi103.stdout:Fetched 57.0 kB in 0s (159 kB/s) 2023-01-24T16:59:36.665 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libffi-dev:amd64. 2023-01-24T16:59:36.724 INFO:teuthology.orchestra.run.smithi103.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152607 files and directories currently installed.) 2023-01-24T16:59:36.729 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../libffi-dev_3.3-4_amd64.deb ... 2023-01-24T16:59:36.805 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libffi-dev:amd64 (3.3-4) ... 2023-01-24T16:59:37.313 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libffi-dev:amd64 (3.3-4) ... 2023-01-24T16:59:37.432 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2023-01-24T16:59:37.976 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-01-24T16:59:39.529 INFO:teuthology.orchestra.run.smithi103.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-01-24T16:59:39.531 INFO:teuthology.packaging:Installing package libsqlite3-dev on ubuntu@smithi103.front.sepia.ceph.com 2023-01-24T16:59:39.532 DEBUG:teuthology.orchestra.run.smithi103:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install libsqlite3-dev 2023-01-24T16:59:39.603 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2023-01-24T16:59:39.793 INFO:teuthology.orchestra.run.smithi103.stdout:Building dependency tree... 2023-01-24T16:59:39.793 INFO:teuthology.orchestra.run.smithi103.stdout:Reading state information... 2023-01-24T16:59:39.966 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages were automatically installed and are no longer required: 2023-01-24T16:59:39.968 INFO:teuthology.orchestra.run.smithi103.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-01-24T16:59:39.968 INFO:teuthology.orchestra.run.smithi103.stdout:Use 'sudo apt autoremove' to remove them. 2023-01-24T16:59:39.968 INFO:teuthology.orchestra.run.smithi103.stdout:The following additional packages will be installed: 2023-01-24T16:59:39.969 INFO:teuthology.orchestra.run.smithi103.stdout: libsqlite3-0 2023-01-24T16:59:39.969 INFO:teuthology.orchestra.run.smithi103.stdout:Suggested packages: 2023-01-24T16:59:39.970 INFO:teuthology.orchestra.run.smithi103.stdout: sqlite3-doc 2023-01-24T16:59:40.003 INFO:teuthology.orchestra.run.smithi103.stdout:The following NEW packages will be installed: 2023-01-24T16:59:40.003 INFO:teuthology.orchestra.run.smithi103.stdout: libsqlite3-dev 2023-01-24T16:59:40.005 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages will be upgraded: 2023-01-24T16:59:40.005 INFO:teuthology.orchestra.run.smithi103.stdout: libsqlite3-0 2023-01-24T16:59:40.063 INFO:teuthology.orchestra.run.smithi103.stdout:1 upgraded, 1 newly installed, 0 to remove and 119 not upgraded. 2023-01-24T16:59:40.063 INFO:teuthology.orchestra.run.smithi103.stdout:Need to get 1,245 kB of archives. 2023-01-24T16:59:40.063 INFO:teuthology.orchestra.run.smithi103.stdout:After this operation, 2,372 kB of additional disk space will be used. 2023-01-24T16:59:40.064 INFO:teuthology.orchestra.run.smithi103.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libsqlite3-0 amd64 3.31.1-4ubuntu0.5 [549 kB] 2023-01-24T16:59:40.176 INFO:teuthology.orchestra.run.smithi103.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libsqlite3-dev amd64 3.31.1-4ubuntu0.5 [697 kB] 2023-01-24T16:59:40.561 INFO:teuthology.orchestra.run.smithi103.stdout:Fetched 1,245 kB in 0s (6,871 kB/s) 2023-01-24T16:59:40.746 INFO:teuthology.orchestra.run.smithi103.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152641 files and directories currently installed.) 2023-01-24T16:59:40.750 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../libsqlite3-0_3.31.1-4ubuntu0.5_amd64.deb ... 2023-01-24T16:59:40.948 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libsqlite3-0:amd64 (3.31.1-4ubuntu0.5) over (3.31.1-4ubuntu0.3) ... 2023-01-24T16:59:41.302 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libsqlite3-dev:amd64. 2023-01-24T16:59:41.319 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../libsqlite3-dev_3.31.1-4ubuntu0.5_amd64.deb ... 2023-01-24T16:59:41.353 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libsqlite3-dev:amd64 (3.31.1-4ubuntu0.5) ... 2023-01-24T16:59:41.728 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libsqlite3-0:amd64 (3.31.1-4ubuntu0.5) ... 2023-01-24T16:59:41.852 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libsqlite3-dev:amd64 (3.31.1-4ubuntu0.5) ... 2023-01-24T16:59:41.979 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-01-24T16:59:42.506 INFO:teuthology.orchestra.run.smithi103.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-01-24T16:59:42.508 INFO:teuthology.packaging:Installing package libssl-dev on ubuntu@smithi103.front.sepia.ceph.com 2023-01-24T16:59:42.508 DEBUG:teuthology.orchestra.run.smithi103:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install libssl-dev 2023-01-24T16:59:42.580 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2023-01-24T16:59:42.768 INFO:teuthology.orchestra.run.smithi103.stdout:Building dependency tree... 2023-01-24T16:59:42.768 INFO:teuthology.orchestra.run.smithi103.stdout:Reading state information... 2023-01-24T16:59:42.935 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages were automatically installed and are no longer required: 2023-01-24T16:59:42.936 INFO:teuthology.orchestra.run.smithi103.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-01-24T16:59:42.936 INFO:teuthology.orchestra.run.smithi103.stdout:Use 'sudo apt autoremove' to remove them. 2023-01-24T16:59:42.937 INFO:teuthology.orchestra.run.smithi103.stdout:Suggested packages: 2023-01-24T16:59:42.937 INFO:teuthology.orchestra.run.smithi103.stdout: libssl-doc 2023-01-24T16:59:42.969 INFO:teuthology.orchestra.run.smithi103.stdout:The following NEW packages will be installed: 2023-01-24T16:59:42.970 INFO:teuthology.orchestra.run.smithi103.stdout: libssl-dev 2023-01-24T16:59:43.151 INFO:teuthology.orchestra.run.smithi103.stdout:0 upgraded, 1 newly installed, 0 to remove and 119 not upgraded. 2023-01-24T16:59:43.151 INFO:teuthology.orchestra.run.smithi103.stdout:Need to get 1,584 kB of archives. 2023-01-24T16:59:43.151 INFO:teuthology.orchestra.run.smithi103.stdout:After this operation, 8,009 kB of additional disk space will be used. 2023-01-24T16:59:43.152 INFO:teuthology.orchestra.run.smithi103.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2023-01-24T16:59:44.084 INFO:teuthology.orchestra.run.smithi103.stdout:Fetched 1,584 kB in 1s (2,113 kB/s) 2023-01-24T16:59:44.184 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-01-24T16:59:44.246 INFO:teuthology.orchestra.run.smithi103.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152650 files and directories currently installed.) 2023-01-24T16:59:44.250 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2023-01-24T16:59:44.299 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2023-01-24T16:59:44.793 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2023-01-24T16:59:45.198 INFO:teuthology.orchestra.run.smithi103.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-01-24T16:59:45.200 INFO:teuthology.packaging:Installing package python-dev on ubuntu@smithi103.front.sepia.ceph.com 2023-01-24T16:59:45.201 DEBUG:teuthology.orchestra.run.smithi103:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y --force-yes install python-dev 2023-01-24T16:59:45.268 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2023-01-24T16:59:45.455 INFO:teuthology.orchestra.run.smithi103.stdout:Building dependency tree... 2023-01-24T16:59:45.456 INFO:teuthology.orchestra.run.smithi103.stdout:Reading state information... 2023-01-24T16:59:45.624 INFO:teuthology.orchestra.run.smithi103.stdout:python-dev-is-python2 is already the newest version (2.7.17-4). 2023-01-24T16:59:45.625 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages were automatically installed and are no longer required: 2023-01-24T16:59:45.626 INFO:teuthology.orchestra.run.smithi103.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-01-24T16:59:45.626 INFO:teuthology.orchestra.run.smithi103.stdout:Use 'sudo apt autoremove' to remove them. 2023-01-24T16:59:45.665 INFO:teuthology.orchestra.run.smithi103.stdout:0 upgraded, 0 newly installed, 0 to remove and 119 not upgraded. 2023-01-24T16:59:45.666 INFO:teuthology.orchestra.run.smithi103.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-01-24T16:59:45.667 DEBUG:teuthology.orchestra.run.smithi103:> cd /home/ubuntu/cephtest/pykmip && cp /home/ubuntu/cephtest/pykmip/examples/policy.json /home/ubuntu/cephtest/pykmip 2023-01-24T16:59:45.673 INFO:tasks.pykmip:#0 cclient='client.0' cconfig={'clientca': 'kmiproot', 'clientcert': 'kmip-client', 'secrets': [{'name': 'pykmip-my-key-1'}, {'name': 'pykmip-my-key-2'}], 'servercert': 'kmip-server'} 2023-01-24T16:59:45.674 INFO:tasks.pykmip:#1 ip,p,h 172.21.15.103 5696 smithi103.front.sepia.ceph.com 2023-01-24T16:59:45.674 INFO:tasks.pykmip:#2 clientca kmiproot 2023-01-24T16:59:45.675 INFO:tasks.pykmip:#3 servercert kmip-server 2023-01-24T16:59:45.675 INFO:tasks.pykmip:#4 servercert Namespace(certificate='/home/ubuntu/cephtest/ca/kmip-server.crt', client='client.0', key='/home/ubuntu/cephtest/ca/kmip-server.key', key_type='rsa:2048', name='kmip-server', remote=Remote(name='ubuntu@smithi103.front.sepia.ceph.com')) 2023-01-24T16:59:45.675 INFO:tasks.pykmip:#3 clientcert kmip-client 2023-01-24T16:59:45.676 INFO:tasks.pykmip:#4 clientcert Namespace(certificate='/home/ubuntu/cephtest/ca/kmip-client.crt', client='client.0', key='/home/ubuntu/cephtest/ca/kmip-client.key', key_type='rsa:2048', name='kmip-client', remote=Remote(name='ubuntu@smithi103.front.sepia.ceph.com')) 2023-01-24T16:59:45.676 INFO:tasks.pykmip:#5 clientca Namespace(certificate='/home/ubuntu/cephtest/ca/kmiproot.crt', client='client.0', key='/home/ubuntu/cephtest/ca/kmiproot.key', key_type='rsa:4096', name='kmiproot', remote=Remote(name='ubuntu@smithi103.front.sepia.ceph.com')) 2023-01-24T16:59:45.676 INFO:tasks.pykmip:#6 serverkey /home/ubuntu/cephtest/ca/kmip-server.key servercert /home/ubuntu/cephtest/ca/kmip-server.crt 2023-01-24T16:59:45.677 INFO:tasks.pykmip:#6 clientkey /home/ubuntu/cephtest/ca/kmip-client.key clientcert /home/ubuntu/cephtest/ca/kmip-client.crt 2023-01-24T16:59:45.677 INFO:tasks.pykmip:#7 clientca /home/ubuntu/cephtest/ca/kmiproot.crt 2023-01-24T16:59:45.687 INFO:tasks.pykmip:Running pykmip... 2023-01-24T16:59:45.687 INFO:tasks.pykmip.client.0:Restarting daemon 2023-01-24T16:59:45.688 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/pykmip && exec bash -c 'cd /home/ubuntu/cephtest/pykmip && . .pykmipenv/bin/activate && HOME=/home/ubuntu/cephtest/pykmip && exec pykmip-server -f pykmip.conf -l /home/ubuntu/cephtest/pykmip/pykmip.log & { read; kill %1; }') 2023-01-24T16:59:45.690 INFO:tasks.pykmip.client.0:Started 2023-01-24T16:59:55.692 DEBUG:teuthology.orchestra.run.smithi103:> . cephtest/pykmip/.pykmipenv/bin/activate;python 2023-01-24T16:59:56.250 INFO:teuthology.orchestra.run.smithi103.stdout:[{"Unique Identifier": "1", "Name": "pykmip-my-key-1", "Object Type": "ObjectType.SYMMETRIC_KEY", "Cryptographic Algorithm": "CryptographicAlgorithm.AES", "Cryptographic Length": "256", "Operation Policy Name": "default", "Cryptographic Usage Mask": "12", "State": "State.ACTIVE", "Initial Date": "Tue Jan 24 16:59:55 2023"}, {"Unique Identifier": "2", "Name": "pykmip-my-key-2", "Object Type": "ObjectType.SYMMETRIC_KEY", "Cryptographic Algorithm": "CryptographicAlgorithm.AES", "Cryptographic Length": "256", "Operation Policy Name": "default", "Cryptographic Usage Mask": "12", "State": "State.ACTIVE", "Initial Date": "Tue Jan 24 16:59:56 2023"}] 2023-01-24T16:59:56.298 INFO:teuthology.run_tasks:Running task rgw... 2023-01-24T16:59:56.313 DEBUG:tasks.rgw:config is {'client.0': {'use-pykmip-role': 'client.0'}} 2023-01-24T16:59:56.314 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2023-01-24T16:59:56.314 INFO:tasks.rgw:Creating data pools 2023-01-24T16:59:56.314 DEBUG:tasks.rgw:Obtaining remote for client client.0 2023-01-24T16:59:56.315 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2023-01-24T16:59:56.785 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'default.rgw.buckets.data' created 2023-01-24T16:59:56.797 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2023-01-24T16:59:57.801 INFO:teuthology.orchestra.run.smithi103.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-01-24T16:59:57.823 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2023-01-24T16:59:58.801 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'default.rgw.buckets.index' created 2023-01-24T16:59:58.811 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2023-01-24T16:59:59.837 INFO:teuthology.orchestra.run.smithi103.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-01-24T16:59:59.848 DEBUG:tasks.rgw:Pools created 2023-01-24T16:59:59.849 INFO:tasks.rgw:Starting rgw... 2023-01-24T16:59:59.849 INFO:tasks.rgw:rgw client.0 config is {'use-pykmip-role': 'client.0'} 2023-01-24T16:59:59.849 INFO:tasks.rgw:Using beast as radosgw frontend 2023-01-24T16:59:59.850 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /home/ubuntu/cephtest/ca/kmip-client.crt /etc/ceph/ 2023-01-24T16:59:59.860 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /home/ubuntu/cephtest/ca/kmip-client.key /etc/ceph/ 2023-01-24T16:59:59.870 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /home/ubuntu/cephtest/ca/kmip-server.crt /etc/ceph/ 2023-01-24T16:59:59.879 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /home/ubuntu/cephtest/ca/kmip-server.key /etc/ceph/ 2023-01-24T16:59:59.889 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /home/ubuntu/cephtest/ca/kmiproot.key /etc/ceph/ 2023-01-24T16:59:59.900 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /home/ubuntu/cephtest/ca/kmiproot.crt /etc/ceph/ 2023-01-24T16:59:59.957 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 600 /etc/ceph/kmip-client.crt /etc/ceph/kmip-client.key /etc/ceph/kmip-server.crt /etc/ceph/kmip-server.key /etc/ceph/kmiproot.key /etc/ceph/kmiproot.crt 2023-01-24T16:59:59.967 DEBUG:teuthology.orchestra.run.smithi103:> sudo chown ceph /etc/ceph/kmip-client.crt /etc/ceph/kmip-client.key /etc/ceph/kmip-server.crt /etc/ceph/kmip-server.key /etc/ceph/kmiproot.key /etc/ceph/kmiproot.crt 2023-01-24T16:59:59.978 INFO:tasks.rgw.client.0:Restarting daemon 2023-01-24T16:59:59.979 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_kmip_addr 172.21.15.103:5696 --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2023-01-24T17:00:00.023 INFO:tasks.rgw.client.0:Started 2023-01-24T17:00:00.023 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi103.front.sepia.ceph.com:80/ 2023-01-24T17:00:00.024 DEBUG:teuthology.orchestra.run.smithi103:> curl http://smithi103.front.sepia.ceph.com:80/ 2023-01-24T17:00:00.062 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-01-24T17:00:00.063 INFO:teuthology.orchestra.run.smithi103.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:00:00.063 INFO:teuthology.orchestra.run.smithi103.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:00:00.063 INFO:teuthology.orchestra.run.smithi103.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-01-24T17:00:00.063 INFO:teuthology.orchestra.run.smithi103.stderr:curl: (7) Failed to connect to smithi103.front.sepia.ceph.com port 80: Connection refused 2023-01-24T17:00:01.064 DEBUG:teuthology.orchestra.run.smithi103:> curl http://smithi103.front.sepia.ceph.com:80/ 2023-01-24T17:00:01.084 INFO:teuthology.orchestra.run.smithi103.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:00:01.084 INFO:teuthology.orchestra.run.smithi103.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:00:01.084 INFO:teuthology.orchestra.run.smithi103.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-01-24T17:00:01.085 INFO:teuthology.orchestra.run.smithi103.stderr:curl: (7) Failed to connect to smithi103.front.sepia.ceph.com port 80: Connection refused 2023-01-24T17:00:01.085 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-01-24T17:00:03.087 DEBUG:teuthology.orchestra.run.smithi103:> curl http://smithi103.front.sepia.ceph.com:80/ 2023-01-24T17:00:03.104 INFO:teuthology.orchestra.run.smithi103.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:00:03.104 INFO:teuthology.orchestra.run.smithi103.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:00:03.105 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-01-24T17:00:03.106 INFO:teuthology.orchestra.run.smithi103.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-01-24T17:00:03.106 INFO:teuthology.orchestra.run.smithi103.stderr:curl: (7) Failed to connect to smithi103.front.sepia.ceph.com port 80: Connection refused 2023-01-24T17:00:07.107 DEBUG:teuthology.orchestra.run.smithi103:> curl http://smithi103.front.sepia.ceph.com:80/ 2023-01-24T17:00:07.124 INFO:teuthology.orchestra.run.smithi103.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:00:07.124 INFO:teuthology.orchestra.run.smithi103.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:00:07.125 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-01-24T17:00:07.126 INFO:teuthology.orchestra.run.smithi103.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-01-24T17:00:07.126 INFO:teuthology.orchestra.run.smithi103.stderr:curl: (7) Failed to connect to smithi103.front.sepia.ceph.com port 80: Connection refused 2023-01-24T17:00:15.127 DEBUG:teuthology.orchestra.run.smithi103:> curl http://smithi103.front.sepia.ceph.com:80/ 2023-01-24T17:00:15.145 INFO:teuthology.orchestra.run.smithi103.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-01-24T17:00:15.145 INFO:teuthology.orchestra.run.smithi103.stderr: Dload Upload Total Spent Left Speed 2023-01-24T17:00:15.149 INFO:teuthology.orchestra.run.smithi103.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-01-24T17:00:15.149 INFO:teuthology.orchestra.run.smithi103.stdout:anonymous 2023-01-24T17:00:15.150 INFO:teuthology.run_tasks:Running task s3tests... 2023-01-24T17:00:15.170 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'barbican': {'kms_key': 'my-key-1', 'kms_key2': 'my-key-2'}, 'force-branch': 'ceph-quincy', 'vault_kv': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}, 'vault_old': {'key_path': 'my-key-1/1', 'key_path2': 'my-key-2/1'}, 'vault_transit': {'key_path': 'my-key-1', 'key_path2': 'my-key-2'}}} 2023-01-24T17:00:15.171 INFO:tasks.s3tests:Downloading s3-tests... 2023-01-24T17:00:15.171 INFO:tasks.s3tests:Using branch 'ceph-quincy' for s3tests 2023-01-24T17:00:15.172 DEBUG:teuthology.orchestra.run.smithi103:> git clone -b ceph-quincy https://git.ceph.com/s3-tests.git /home/ubuntu/cephtest/s3-tests 2023-01-24T17:00:15.196 INFO:teuthology.orchestra.run.smithi103.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests'... 2023-01-24T17:00:15.934 INFO:tasks.s3tests:Creating rgw users... 2023-01-24T17:00:15.935 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2023-01-24T17:00:15.935 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key GMDOQETASXBLZQIFOCKN --secret TugGcyK29M9HI7tmd2dPXXi1AKQR/koBt6rIQqVzbVwqqbx8zKaWWA== --email foo.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-01-24T17:00:16.024 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:00:16.024 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:00:16.035 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.026+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.030+0000 7f2f3f764e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.030+0000 7f2f3f764e40 20 realm 2023-01-24T17:00:16.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.030+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.039 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.030+0000 7f2f3f764e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.039 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.030+0000 7f2f3f764e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:00:16.039 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.030+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.039 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.030+0000 7f2f3f764e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.040 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.030+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.040 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.030+0000 7f2f3f764e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:16.040 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.030+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.041 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.030+0000 7f2f3f764e40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:00:16.041 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.030+0000 7f2f3f764e40 20 searching for the correct realm 2023-01-24T17:00:16.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.050+0000 7f2f3f764e40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:00:16.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.050+0000 7f2f3f764e40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:00:16.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.050+0000 7f2f3f764e40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:00:16.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.050+0000 7f2f3f764e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:00:16.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.050+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.050+0000 7f2f3f764e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.050+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.050+0000 7f2f3f764e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:16.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.050+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.050+0000 7f2f3f764e40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:00:16.061 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.050+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.050+0000 7f2f3f764e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.050+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.050+0000 7f2f3f764e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.062 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.050+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.063 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.054+0000 7f2f3f764e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:16.082 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:00:16.083 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:00:16.083 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:00:16.084 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:00:16.084 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.085 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:16.085 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.085 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:00:16.086 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.086 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:16.086 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.086 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:00:16.086 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.087 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.087 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 zone default found 2023-01-24T17:00:16.087 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 4 Realm: () 2023-01-24T17:00:16.087 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 4 ZoneGroup: default (0d4b5768-74c8-4719-bc1b-4550aa389781) 2023-01-24T17:00:16.088 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 4 Zone: default (2ee3967b-537d-4521-a85e-a66f0a1f41f2) 2023-01-24T17:00:16.088 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:00:16.088 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 zonegroup default 2023-01-24T17:00:16.088 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.074+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.078+0000 7f2f3f764e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.078+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.090 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.078+0000 7f2f3f764e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.090 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.078+0000 7f2f3f764e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.090 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.078+0000 7f2f3f764e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.090 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.078+0000 7f2f3f764e40 20 started sync module instance, tier type = 2023-01-24T17:00:16.090 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.078+0000 7f2f3f764e40 20 started zone id=2ee3967b-537d-4521-a85e-a66f0a1f41f2 (name=default) with tier type = 2023-01-24T17:00:16.107 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.098+0000 7f2f3f764e40 20 add_watcher() i=0 2023-01-24T17:00:16.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.098+0000 7f2f3f764e40 20 add_watcher() i=1 2023-01-24T17:00:16.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.098+0000 7f2f3f764e40 20 add_watcher() i=2 2023-01-24T17:00:16.108 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.098+0000 7f2f3f764e40 20 add_watcher() i=3 2023-01-24T17:00:16.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.098+0000 7f2f3f764e40 20 add_watcher() i=4 2023-01-24T17:00:16.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.098+0000 7f2f3f764e40 20 add_watcher() i=5 2023-01-24T17:00:16.109 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.098+0000 7f2f3f764e40 20 add_watcher() i=6 2023-01-24T17:00:16.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.098+0000 7f2f3f764e40 20 add_watcher() i=7 2023-01-24T17:00:16.110 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.098+0000 7f2f3f764e40 2 all 8 watchers are set, enabling cache 2023-01-24T17:00:16.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.102+0000 7f2f0e7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:00:16.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.102+0000 7f2f3f764e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:00:16.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.102+0000 7f2f3f764e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:00:16.118 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.110+0000 7f2f3f764e40 5 note: GC not initialized 2023-01-24T17:00:16.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.110+0000 7f2e7d7fa700 20 reqs_thread_entry: start 2023-01-24T17:00:16.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.110+0000 7f2e76ffd700 20 reqs_thread_entry: start 2023-01-24T17:00:16.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.110+0000 7f2f3f764e40 20 init_complete bucket index max shards: 11 2023-01-24T17:00:16.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:00:16.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2e757fa700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:00:46 2023 2023-01-24T17:00:16.122 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-01-24T17:00:16.123 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2e74ff9700 20 reqs_thread_entry: start 2023-01-24T17:00:16.123 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 20 get_system_obj_state: rctx=0x7fff902551b8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55c54f929ef0 s->prefetch_data=0 2023-01-24T17:00:16.123 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-01-24T17:00:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2023-01-24T17:00:16.124 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:00:16.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 20 get_system_obj_state: rctx=0x7fff90254ea8 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x55c54f929ef0 s->prefetch_data=0 2023-01-24T17:00:16.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2023-01-24T17:00:16.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2023-01-24T17:00:16.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-01-24T17:00:16.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 20 get_system_obj_state: rctx=0x7fff90254ea8 obj=default.rgw.meta:users.keys:GMDOQETASXBLZQIFOCKN state=0x55c54f929ef0 s->prefetch_data=0 2023-01-24T17:00:16.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 10 cache get: name=default.rgw.meta+users.keys+GMDOQETASXBLZQIFOCKN : miss 2023-01-24T17:00:16.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 10 cache put: name=default.rgw.meta+users.keys+GMDOQETASXBLZQIFOCKN info.flags=0x0 2023-01-24T17:00:16.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 10 adding default.rgw.meta+users.keys+GMDOQETASXBLZQIFOCKN to cache LRU end 2023-01-24T17:00:16.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 20 get_system_obj_state: rctx=0x7fff90254748 obj=default.rgw.meta:users.keys:GMDOQETASXBLZQIFOCKN state=0x55c54f929ef0 s->prefetch_data=0 2023-01-24T17:00:16.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 10 cache get: name=default.rgw.meta+users.keys+GMDOQETASXBLZQIFOCKN : hit (negative entry) 2023-01-24T17:00:16.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 20 get_system_obj_state: rctx=0x7fff90254de8 obj=default.rgw.meta:users.keys:GMDOQETASXBLZQIFOCKN state=0x55c54f929ef0 s->prefetch_data=0 2023-01-24T17:00:16.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.114+0000 7f2f3f764e40 10 cache get: name=default.rgw.meta+users.keys+GMDOQETASXBLZQIFOCKN : hit (negative entry) 2023-01-24T17:00:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.118+0000 7f2f3f764e40 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-01-24T17:00:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.118+0000 7f2f3f764e40 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:00:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.118+0000 7f2f3f764e40 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-01-24T17:00:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.118+0000 7f2f36ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084288 cookie 94305933391952 notifier 4211 bl.length()=640 2023-01-24T17:00:16.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.118+0000 7f2f36ffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-01-24T17:00:16.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.118+0000 7f2f36ffd700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:00:16.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.118+0000 7f2f3f764e40 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-01-24T17:00:16.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.118+0000 7f2f3f764e40 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-01-24T17:00:16.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.118+0000 7f2f3f764e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-01-24T17:00:16.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.122+0000 7f2f367fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084288 cookie 94305933408880 notifier 4211 bl.length()=194 2023-01-24T17:00:16.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.122+0000 7f2f367fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2023-01-24T17:00:16.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.122+0000 7f2f367fc700 10 rgw watcher librados: moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2023-01-24T17:00:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.122+0000 7f2f3f764e40 10 cache put: name=default.rgw.meta+users.keys+GMDOQETASXBLZQIFOCKN info.flags=0x7 2023-01-24T17:00:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.122+0000 7f2f3f764e40 10 moving default.rgw.meta+users.keys+GMDOQETASXBLZQIFOCKN to cache LRU end 2023-01-24T17:00:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.122+0000 7f2f3f764e40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.keys:GMDOQETASXBLZQIFOCKN, ofs0, ns] 2023-01-24T17:00:16.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.126+0000 7f2f36ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 120259084289 cookie 94305933410544 notifier 4211 bl.length()=186 2023-01-24T17:00:16.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.126+0000 7f2f36ffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+GMDOQETASXBLZQIFOCKN info.flags=0x7 2023-01-24T17:00:16.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.126+0000 7f2f36ffd700 10 rgw watcher librados: moving default.rgw.meta+users.keys+GMDOQETASXBLZQIFOCKN to cache LRU end 2023-01-24T17:00:16.135 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2023-01-24T17:00:16.135 INFO:teuthology.orchestra.run.smithi103.stdout: "user_id": "foo.client.0", 2023-01-24T17:00:16.135 INFO:teuthology.orchestra.run.smithi103.stdout: "display_name": "Mr. foo.client.0", 2023-01-24T17:00:16.136 INFO:teuthology.orchestra.run.smithi103.stdout: "email": "foo.client.0+test@test.test", 2023-01-24T17:00:16.136 INFO:teuthology.orchestra.run.smithi103.stdout: "suspended": 0, 2023-01-24T17:00:16.136 INFO:teuthology.orchestra.run.smithi103.stdout: "max_buckets": 1000, 2023-01-24T17:00:16.137 INFO:teuthology.orchestra.run.smithi103.stdout: "subusers": [], 2023-01-24T17:00:16.137 INFO:teuthology.orchestra.run.smithi103.stdout: "keys": [ 2023-01-24T17:00:16.137 INFO:teuthology.orchestra.run.smithi103.stdout: { 2023-01-24T17:00:16.137 INFO:teuthology.orchestra.run.smithi103.stdout: "user": "foo.client.0", 2023-01-24T17:00:16.137 INFO:teuthology.orchestra.run.smithi103.stdout: "access_key": "GMDOQETASXBLZQIFOCKN", 2023-01-24T17:00:16.137 INFO:teuthology.orchestra.run.smithi103.stdout: "secret_key": "TugGcyK29M9HI7tmd2dPXXi1AKQR/koBt6rIQqVzbVwqqbx8zKaWWA==" 2023-01-24T17:00:16.138 INFO:teuthology.orchestra.run.smithi103.stdout: } 2023-01-24T17:00:16.138 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2023-01-24T17:00:16.138 INFO:teuthology.orchestra.run.smithi103.stdout: "swift_keys": [], 2023-01-24T17:00:16.138 INFO:teuthology.orchestra.run.smithi103.stdout: "caps": [ 2023-01-24T17:00:16.138 INFO:teuthology.orchestra.run.smithi103.stdout: { 2023-01-24T17:00:16.139 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "user-policy", 2023-01-24T17:00:16.139 INFO:teuthology.orchestra.run.smithi103.stdout: "perm": "*" 2023-01-24T17:00:16.139 INFO:teuthology.orchestra.run.smithi103.stdout: } 2023-01-24T17:00:16.139 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2023-01-24T17:00:16.139 INFO:teuthology.orchestra.run.smithi103.stdout: "op_mask": "read, write, delete", 2023-01-24T17:00:16.140 INFO:teuthology.orchestra.run.smithi103.stdout: "default_placement": "", 2023-01-24T17:00:16.140 INFO:teuthology.orchestra.run.smithi103.stdout: "default_storage_class": "", 2023-01-24T17:00:16.140 INFO:teuthology.orchestra.run.smithi103.stdout: "placement_tags": [], 2023-01-24T17:00:16.140 INFO:teuthology.orchestra.run.smithi103.stdout: "bucket_quota": { 2023-01-24T17:00:16.140 INFO:teuthology.orchestra.run.smithi103.stdout: "enabled": false, 2023-01-24T17:00:16.141 INFO:teuthology.orchestra.run.smithi103.stdout: "check_on_raw": false, 2023-01-24T17:00:16.141 INFO:teuthology.orchestra.run.smithi103.stdout: "max_size": -1, 2023-01-24T17:00:16.141 INFO:teuthology.orchestra.run.smithi103.stdout: "max_size_kb": 0, 2023-01-24T17:00:16.141 INFO:teuthology.orchestra.run.smithi103.stdout: "max_objects": -1 2023-01-24T17:00:16.141 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2023-01-24T17:00:16.141 INFO:teuthology.orchestra.run.smithi103.stdout: "user_quota": { 2023-01-24T17:00:16.142 INFO:teuthology.orchestra.run.smithi103.stdout: "enabled": false, 2023-01-24T17:00:16.142 INFO:teuthology.orchestra.run.smithi103.stdout: "check_on_raw": false, 2023-01-24T17:00:16.142 INFO:teuthology.orchestra.run.smithi103.stdout: "max_size": -1, 2023-01-24T17:00:16.142 INFO:teuthology.orchestra.run.smithi103.stdout: "max_size_kb": 0, 2023-01-24T17:00:16.142 INFO:teuthology.orchestra.run.smithi103.stdout: "max_objects": -1 2023-01-24T17:00:16.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2023-01-24T17:00:16.143 INFO:teuthology.orchestra.run.smithi103.stdout: "temp_url_keys": [], 2023-01-24T17:00:16.143 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "rgw", 2023-01-24T17:00:16.143 INFO:teuthology.orchestra.run.smithi103.stdout: "mfa_ids": [] 2023-01-24T17:00:16.143 INFO:teuthology.orchestra.run.smithi103.stdout:} 2023-01-24T17:00:16.143 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:16.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.130+0000 7f2f3f764e40 20 remove_watcher() i=0 2023-01-24T17:00:16.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.130+0000 7f2f3f764e40 2 removed watcher, disabling cache 2023-01-24T17:00:16.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.130+0000 7f2f3f764e40 20 remove_watcher() i=1 2023-01-24T17:00:16.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.134+0000 7f2f3f764e40 20 remove_watcher() i=2 2023-01-24T17:00:16.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.134+0000 7f2f3f764e40 20 remove_watcher() i=3 2023-01-24T17:00:16.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.134+0000 7f2f3f764e40 20 remove_watcher() i=4 2023-01-24T17:00:16.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.138+0000 7f2f3f764e40 20 remove_watcher() i=5 2023-01-24T17:00:16.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.142+0000 7f2f3f764e40 20 remove_watcher() i=6 2023-01-24T17:00:16.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.142+0000 7f2f3f764e40 20 remove_watcher() i=7 2023-01-24T17:00:16.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.142+0000 7f2f0e7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:00:16.162 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 3126436219 --totp-seed D34SOQBPMMJTRFMIXLUMEGHNEC3SASKH5VSFKT73DV6JGVPBFI5MGAR5AKLLGRIC --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-01-24T17:00:16.207 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:00:16.208 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:00:16.219 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.210+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.210+0000 7f54681a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.210+0000 7f54681a9e40 20 realm 2023-01-24T17:00:16.221 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.210+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.210+0000 7f54681a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.210+0000 7f54681a9e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:00:16.222 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.210+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.214+0000 7f54681a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.214+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.214+0000 7f54681a9e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:16.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.214+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.214+0000 7f54681a9e40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:00:16.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.214+0000 7f54681a9e40 20 searching for the correct realm 2023-01-24T17:00:16.244 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.234+0000 7f54681a9e40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:00:16.245 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.234+0000 7f54681a9e40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:00:16.245 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.234+0000 7f54681a9e40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:00:16.245 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.234+0000 7f54681a9e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:00:16.245 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.234+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.246 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.234+0000 7f54681a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.247 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.234+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.247 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.238+0000 7f54681a9e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:16.247 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.238+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.247 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.238+0000 7f54681a9e40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:00:16.247 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.238+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.238+0000 7f54681a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.238+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.248 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.238+0000 7f54681a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.238+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.238+0000 7f54681a9e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:16.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.262+0000 7f54681a9e40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:00:16.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.262+0000 7f54681a9e40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:00:16.272 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.262+0000 7f54681a9e40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:00:16.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.262+0000 7f54681a9e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:00:16.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.262+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.262+0000 7f54681a9e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:16.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.262+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.262+0000 7f54681a9e40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:00:16.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.262+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.262+0000 7f54681a9e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:16.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.262+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.275 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:00:16.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 20 zone default found 2023-01-24T17:00:16.277 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 4 Realm: () 2023-01-24T17:00:16.277 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 4 ZoneGroup: default (0d4b5768-74c8-4719-bc1b-4550aa389781) 2023-01-24T17:00:16.277 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 4 Zone: default (2ee3967b-537d-4521-a85e-a66f0a1f41f2) 2023-01-24T17:00:16.277 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:00:16.277 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 20 zonegroup default 2023-01-24T17:00:16.278 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.278 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.279 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.279 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.279 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.280 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:16.280 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 20 started sync module instance, tier type = 2023-01-24T17:00:16.280 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.266+0000 7f54681a9e40 20 started zone id=2ee3967b-537d-4521-a85e-a66f0a1f41f2 (name=default) with tier type = 2023-01-24T17:00:16.302 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.294+0000 7f54681a9e40 20 add_watcher() i=0 2023-01-24T17:00:16.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.294+0000 7f54681a9e40 20 add_watcher() i=1 2023-01-24T17:00:16.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.294+0000 7f54681a9e40 20 add_watcher() i=2 2023-01-24T17:00:16.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.294+0000 7f54681a9e40 20 add_watcher() i=3 2023-01-24T17:00:16.303 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.294+0000 7f54681a9e40 20 add_watcher() i=4 2023-01-24T17:00:16.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.294+0000 7f54681a9e40 20 add_watcher() i=5 2023-01-24T17:00:16.304 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.294+0000 7f54681a9e40 20 add_watcher() i=6 2023-01-24T17:00:16.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.294+0000 7f54681a9e40 20 add_watcher() i=7 2023-01-24T17:00:16.305 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.294+0000 7f54681a9e40 2 all 8 watchers are set, enabling cache 2023-01-24T17:00:16.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.298+0000 7f5436fe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:00:16.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.298+0000 7f54681a9e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:00:16.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.298+0000 7f54681a9e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:00:16.317 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.306+0000 7f54681a9e40 5 note: GC not initialized 2023-01-24T17:00:16.319 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.310+0000 7f53a5ffb700 20 reqs_thread_entry: start 2023-01-24T17:00:16.321 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.310+0000 7f54681a9e40 20 init_complete bucket index max shards: 11 2023-01-24T17:00:16.321 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.310+0000 7f53977fe700 20 reqs_thread_entry: start 2023-01-24T17:00:16.321 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.310+0000 7f54681a9e40 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:00:16.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.310+0000 7f5395ffb700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:00:46 2023 2023-01-24T17:00:16.322 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-01-24T17:00:16.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.310+0000 7f53957fa700 20 reqs_thread_entry: start 2023-01-24T17:00:16.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.310+0000 7f54681a9e40 20 get_system_obj_state: rctx=0x7ffc2eb599a8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55e457256ef0 s->prefetch_data=0 2023-01-24T17:00:16.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.310+0000 7f54681a9e40 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-01-24T17:00:16.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.310+0000 7f54681a9e40 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-01-24T17:00:16.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.310+0000 7f54681a9e40 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:00:16.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.310+0000 7f54681a9e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:00:16.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.310+0000 7f54681a9e40 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:00:16.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.310+0000 7f54681a9e40 20 Read xattr: user.rgw.idtag 2023-01-24T17:00:16.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.310+0000 7f54681a9e40 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:00:16.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.310+0000 7f54681a9e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:16.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.314+0000 7f54681a9e40 20 rados_obj.operate() r=0 bl.length=455 2023-01-24T17:00:16.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.314+0000 7f54681a9e40 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-01-24T17:00:16.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.314+0000 7f54681a9e40 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:00:16.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:16.314+0000 7f54681a9e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:00:17.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.746+0000 7f54681a9e40 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-01-24T17:00:17.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.746+0000 7f54681a9e40 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:00:17.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.746+0000 7f54681a9e40 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-01-24T17:00:17.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.750+0000 7f5456ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018882 cookie 94439204440704 notifier 4217 bl.length()=654 2023-01-24T17:00:17.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.750+0000 7f5456ffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2023-01-24T17:00:17.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.750+0000 7f5456ffd700 10 rgw watcher librados: moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:00:17.759 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.750+0000 7f54681a9e40 20 remove_watcher() i=0 2023-01-24T17:00:17.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.750+0000 7f54681a9e40 2 removed watcher, disabling cache 2023-01-24T17:00:17.761 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.750+0000 7f54681a9e40 20 remove_watcher() i=1 2023-01-24T17:00:17.762 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.754+0000 7f54681a9e40 20 remove_watcher() i=2 2023-01-24T17:00:17.764 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.754+0000 7f54681a9e40 20 remove_watcher() i=3 2023-01-24T17:00:17.765 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.758+0000 7f54681a9e40 20 remove_watcher() i=4 2023-01-24T17:00:17.769 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.758+0000 7f54681a9e40 20 remove_watcher() i=5 2023-01-24T17:00:17.771 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.762+0000 7f54681a9e40 20 remove_watcher() i=6 2023-01-24T17:00:17.772 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.762+0000 7f54681a9e40 20 remove_watcher() i=7 2023-01-24T17:00:17.774 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.766+0000 7f5436fe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:00:17.783 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2023-01-24T17:00:17.784 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --access-key NJIHOWOUGIVKLZCJLGCP --secret bDntc8tN+59SHs4+Luqb4gDZEb0YMY2JmIvX3JdSAYl1t7aBp1HhyA== --email bar.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2023-01-24T17:00:17.827 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:00:17.828 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:00:17.839 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.830+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.841 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.834+0000 7ff84effde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:17.842 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.834+0000 7ff84effde40 20 realm 2023-01-24T17:00:17.842 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.834+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.842 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.834+0000 7ff84effde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:17.842 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.834+0000 7ff84effde40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:00:17.842 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.834+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.843 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.834+0000 7ff84effde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:17.843 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.834+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.834+0000 7ff84effde40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:17.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.834+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.844 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.834+0000 7ff84effde40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:00:17.845 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.834+0000 7ff84effde40 20 searching for the correct realm 2023-01-24T17:00:17.861 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.850+0000 7ff84effde40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:00:17.862 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.850+0000 7ff84effde40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:00:17.862 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.850+0000 7ff84effde40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:00:17.862 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.850+0000 7ff84effde40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:00:17.862 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.850+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.862 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.850+0000 7ff84effde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:17.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.850+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.850+0000 7ff84effde40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:17.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.850+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.850+0000 7ff84effde40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:00:17.863 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.850+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.850+0000 7ff84effde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:17.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.850+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.850+0000 7ff84effde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:17.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.850+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.864 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.854+0000 7ff84effde40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:17.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.870+0000 7ff84effde40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:00:17.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.870+0000 7ff84effde40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:00:17.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.870+0000 7ff84effde40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:00:17.880 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.870+0000 7ff84effde40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:00:17.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.870+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.870+0000 7ff84effde40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:17.881 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.870+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.882 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.870+0000 7ff84effde40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:00:17.882 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.870+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.882 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.870+0000 7ff84effde40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:17.883 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.883 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:00:17.883 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.884 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:17.884 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 20 zone default found 2023-01-24T17:00:17.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 4 Realm: () 2023-01-24T17:00:17.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 4 ZoneGroup: default (0d4b5768-74c8-4719-bc1b-4550aa389781) 2023-01-24T17:00:17.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 4 Zone: default (2ee3967b-537d-4521-a85e-a66f0a1f41f2) 2023-01-24T17:00:17.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:00:17.885 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 20 zonegroup default 2023-01-24T17:00:17.886 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.886 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:17.886 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.887 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:17.887 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:17.888 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:17.888 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 20 started sync module instance, tier type = 2023-01-24T17:00:17.888 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.874+0000 7ff84effde40 20 started zone id=2ee3967b-537d-4521-a85e-a66f0a1f41f2 (name=default) with tier type = 2023-01-24T17:00:17.916 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.906+0000 7ff84effde40 20 add_watcher() i=0 2023-01-24T17:00:17.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.906+0000 7ff84effde40 20 add_watcher() i=1 2023-01-24T17:00:17.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.906+0000 7ff84effde40 20 add_watcher() i=2 2023-01-24T17:00:17.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.906+0000 7ff84effde40 20 add_watcher() i=3 2023-01-24T17:00:17.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.906+0000 7ff84effde40 20 add_watcher() i=4 2023-01-24T17:00:17.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.906+0000 7ff84effde40 20 add_watcher() i=5 2023-01-24T17:00:17.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.906+0000 7ff84effde40 20 add_watcher() i=6 2023-01-24T17:00:17.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.910+0000 7ff84effde40 20 add_watcher() i=7 2023-01-24T17:00:17.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.910+0000 7ff84effde40 2 all 8 watchers are set, enabling cache 2023-01-24T17:00:17.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.910+0000 7ff81e7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:00:17.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.910+0000 7ff84effde40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:00:17.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.910+0000 7ff84effde40 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:00:17.930 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.922+0000 7ff84effde40 5 note: GC not initialized 2023-01-24T17:00:17.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.922+0000 7ff78d7fa700 20 reqs_thread_entry: start 2023-01-24T17:00:17.934 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff786ffd700 20 reqs_thread_entry: start 2023-01-24T17:00:17.934 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 20 init_complete bucket index max shards: 11 2023-01-24T17:00:17.935 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:00:17.935 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff7857fa700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:00:48 2023 2023-01-24T17:00:17.935 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-01-24T17:00:17.935 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff784ff9700 20 reqs_thread_entry: start 2023-01-24T17:00:17.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 20 get_system_obj_state: rctx=0x7fff88f7b498 obj=default.rgw.meta:users.uid:bar.client.0 state=0x564b3cc34ef0 s->prefetch_data=0 2023-01-24T17:00:17.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-01-24T17:00:17.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2023-01-24T17:00:17.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:00:17.936 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 20 get_system_obj_state: rctx=0x7fff88f7b188 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x564b3cc34ef0 s->prefetch_data=0 2023-01-24T17:00:17.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2023-01-24T17:00:17.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2023-01-24T17:00:17.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-01-24T17:00:17.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 20 get_system_obj_state: rctx=0x7fff88f7b188 obj=default.rgw.meta:users.keys:NJIHOWOUGIVKLZCJLGCP state=0x564b3cc34ef0 s->prefetch_data=0 2023-01-24T17:00:17.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 10 cache get: name=default.rgw.meta+users.keys+NJIHOWOUGIVKLZCJLGCP : miss 2023-01-24T17:00:17.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 10 cache put: name=default.rgw.meta+users.keys+NJIHOWOUGIVKLZCJLGCP info.flags=0x0 2023-01-24T17:00:17.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 10 adding default.rgw.meta+users.keys+NJIHOWOUGIVKLZCJLGCP to cache LRU end 2023-01-24T17:00:17.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 20 get_system_obj_state: rctx=0x7fff88f7aa28 obj=default.rgw.meta:users.keys:NJIHOWOUGIVKLZCJLGCP state=0x564b3cc34ef0 s->prefetch_data=0 2023-01-24T17:00:17.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 10 cache get: name=default.rgw.meta+users.keys+NJIHOWOUGIVKLZCJLGCP : hit (negative entry) 2023-01-24T17:00:17.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 20 get_system_obj_state: rctx=0x7fff88f7b0c8 obj=default.rgw.meta:users.keys:NJIHOWOUGIVKLZCJLGCP state=0x564b3cc34ef0 s->prefetch_data=0 2023-01-24T17:00:17.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.926+0000 7ff84effde40 10 cache get: name=default.rgw.meta+users.keys+NJIHOWOUGIVKLZCJLGCP : hit (negative entry) 2023-01-24T17:00:17.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.930+0000 7ff84effde40 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-01-24T17:00:17.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.930+0000 7ff84effde40 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:00:17.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.930+0000 7ff84effde40 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-01-24T17:00:17.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.930+0000 7ff846ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018881 cookie 94881143427056 notifier 4223 bl.length()=640 2023-01-24T17:00:17.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.930+0000 7ff846ffd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-01-24T17:00:17.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.930+0000 7ff846ffd700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:00:17.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.934+0000 7ff84effde40 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-01-24T17:00:17.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.934+0000 7ff84effde40 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-01-24T17:00:17.943 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.934+0000 7ff84effde40 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-01-24T17:00:17.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.934+0000 7ff8467fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018882 cookie 94881143427056 notifier 4223 bl.length()=194 2023-01-24T17:00:17.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.934+0000 7ff8467fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2023-01-24T17:00:17.944 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.934+0000 7ff8467fc700 10 rgw watcher librados: moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2023-01-24T17:00:17.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.938+0000 7ff84effde40 10 cache put: name=default.rgw.meta+users.keys+NJIHOWOUGIVKLZCJLGCP info.flags=0x7 2023-01-24T17:00:17.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.938+0000 7ff84effde40 10 moving default.rgw.meta+users.keys+NJIHOWOUGIVKLZCJLGCP to cache LRU end 2023-01-24T17:00:17.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.938+0000 7ff84effde40 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.keys:NJIHOWOUGIVKLZCJLGCP, ofs0, ns] 2023-01-24T17:00:17.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.938+0000 7ff8467fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018880 cookie 94881143431840 notifier 4223 bl.length()=186 2023-01-24T17:00:17.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.938+0000 7ff8467fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+NJIHOWOUGIVKLZCJLGCP info.flags=0x7 2023-01-24T17:00:17.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.938+0000 7ff8467fc700 10 rgw watcher librados: moving default.rgw.meta+users.keys+NJIHOWOUGIVKLZCJLGCP to cache LRU end 2023-01-24T17:00:17.948 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2023-01-24T17:00:17.948 INFO:teuthology.orchestra.run.smithi103.stdout: "user_id": "bar.client.0", 2023-01-24T17:00:17.949 INFO:teuthology.orchestra.run.smithi103.stdout: "display_name": "Mr. bar.client.0", 2023-01-24T17:00:17.949 INFO:teuthology.orchestra.run.smithi103.stdout: "email": "bar.client.0+test@test.test", 2023-01-24T17:00:17.949 INFO:teuthology.orchestra.run.smithi103.stdout: "suspended": 0, 2023-01-24T17:00:17.950 INFO:teuthology.orchestra.run.smithi103.stdout: "max_buckets": 1000, 2023-01-24T17:00:17.950 INFO:teuthology.orchestra.run.smithi103.stdout: "subusers": [], 2023-01-24T17:00:17.950 INFO:teuthology.orchestra.run.smithi103.stdout: "keys": [ 2023-01-24T17:00:17.950 INFO:teuthology.orchestra.run.smithi103.stdout: { 2023-01-24T17:00:17.950 INFO:teuthology.orchestra.run.smithi103.stdout: "user": "bar.client.0", 2023-01-24T17:00:17.951 INFO:teuthology.orchestra.run.smithi103.stdout: "access_key": "NJIHOWOUGIVKLZCJLGCP", 2023-01-24T17:00:17.951 INFO:teuthology.orchestra.run.smithi103.stdout: "secret_key": "bDntc8tN+59SHs4+Luqb4gDZEb0YMY2JmIvX3JdSAYl1t7aBp1HhyA==" 2023-01-24T17:00:17.951 INFO:teuthology.orchestra.run.smithi103.stdout: } 2023-01-24T17:00:17.951 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2023-01-24T17:00:17.951 INFO:teuthology.orchestra.run.smithi103.stdout: "swift_keys": [], 2023-01-24T17:00:17.952 INFO:teuthology.orchestra.run.smithi103.stdout: "caps": [ 2023-01-24T17:00:17.952 INFO:teuthology.orchestra.run.smithi103.stdout: { 2023-01-24T17:00:17.952 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "user-policy", 2023-01-24T17:00:17.952 INFO:teuthology.orchestra.run.smithi103.stdout: "perm": "*" 2023-01-24T17:00:17.952 INFO:teuthology.orchestra.run.smithi103.stdout: } 2023-01-24T17:00:17.953 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2023-01-24T17:00:17.953 INFO:teuthology.orchestra.run.smithi103.stdout: "op_mask": "read, write, delete", 2023-01-24T17:00:17.953 INFO:teuthology.orchestra.run.smithi103.stdout: "default_placement": "", 2023-01-24T17:00:17.953 INFO:teuthology.orchestra.run.smithi103.stdout: "default_storage_class": "", 2023-01-24T17:00:17.953 INFO:teuthology.orchestra.run.smithi103.stdout: "placement_tags": [], 2023-01-24T17:00:17.953 INFO:teuthology.orchestra.run.smithi103.stdout: "bucket_quota": { 2023-01-24T17:00:17.954 INFO:teuthology.orchestra.run.smithi103.stdout: "enabled": false, 2023-01-24T17:00:17.954 INFO:teuthology.orchestra.run.smithi103.stdout: "check_on_raw": false, 2023-01-24T17:00:17.954 INFO:teuthology.orchestra.run.smithi103.stdout: "max_size": -1, 2023-01-24T17:00:17.954 INFO:teuthology.orchestra.run.smithi103.stdout: "max_size_kb": 0, 2023-01-24T17:00:17.954 INFO:teuthology.orchestra.run.smithi103.stdout: "max_objects": -1 2023-01-24T17:00:17.955 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2023-01-24T17:00:17.955 INFO:teuthology.orchestra.run.smithi103.stdout: "user_quota": { 2023-01-24T17:00:17.955 INFO:teuthology.orchestra.run.smithi103.stdout: "enabled": false, 2023-01-24T17:00:17.955 INFO:teuthology.orchestra.run.smithi103.stdout: "check_on_raw": false, 2023-01-24T17:00:17.955 INFO:teuthology.orchestra.run.smithi103.stdout: "max_size": -1, 2023-01-24T17:00:17.956 INFO:teuthology.orchestra.run.smithi103.stdout: "max_size_kb": 0, 2023-01-24T17:00:17.956 INFO:teuthology.orchestra.run.smithi103.stdout: "max_objects": -1 2023-01-24T17:00:17.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2023-01-24T17:00:17.956 INFO:teuthology.orchestra.run.smithi103.stdout: "temp_url_keys": [], 2023-01-24T17:00:17.956 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "rgw", 2023-01-24T17:00:17.957 INFO:teuthology.orchestra.run.smithi103.stdout: "mfa_ids": [] 2023-01-24T17:00:17.957 INFO:teuthology.orchestra.run.smithi103.stdout:} 2023-01-24T17:00:17.957 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:17.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.942+0000 7ff84effde40 20 remove_watcher() i=0 2023-01-24T17:00:17.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.942+0000 7ff84effde40 2 removed watcher, disabling cache 2023-01-24T17:00:17.958 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.942+0000 7ff84effde40 20 remove_watcher() i=1 2023-01-24T17:00:17.959 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.946+0000 7ff84effde40 20 remove_watcher() i=2 2023-01-24T17:00:17.959 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.950+0000 7ff84effde40 20 remove_watcher() i=3 2023-01-24T17:00:17.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.950+0000 7ff84effde40 20 remove_watcher() i=4 2023-01-24T17:00:17.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.954+0000 7ff84effde40 20 remove_watcher() i=5 2023-01-24T17:00:17.964 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.954+0000 7ff84effde40 20 remove_watcher() i=6 2023-01-24T17:00:17.966 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.958+0000 7ff84effde40 20 remove_watcher() i=7 2023-01-24T17:00:17.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:17.958+0000 7ff81e7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:00:17.977 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 6531764678 --totp-seed AIOLQJVXO4P6JAN5ZWL2AFRHW4S57TVH2ZD5LX5JZIMAFJVEGQG4WPZSNNNRVL66 --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-01-24T17:00:18.025 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:00:18.026 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:00:18.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.026+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.040 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.030+0000 7f2b69a13e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.040 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.030+0000 7f2b69a13e40 20 realm 2023-01-24T17:00:18.040 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.030+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.041 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.030+0000 7f2b69a13e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.041 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.030+0000 7f2b69a13e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:00:18.041 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.030+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.041 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.030+0000 7f2b69a13e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.041 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.030+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.042 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.034+0000 7f2b69a13e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:18.042 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.034+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.043 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.034+0000 7f2b69a13e40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:00:18.043 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.034+0000 7f2b69a13e40 20 searching for the correct realm 2023-01-24T17:00:18.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.054+0000 7f2b69a13e40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:00:18.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.054+0000 7f2b69a13e40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:00:18.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.054+0000 7f2b69a13e40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:00:18.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.054+0000 7f2b69a13e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:00:18.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.054+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.065 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.054+0000 7f2b69a13e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.054+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.054+0000 7f2b69a13e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:18.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.054+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.058+0000 7f2b69a13e40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:00:18.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.058+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.068 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.058+0000 7f2b69a13e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.068 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.058+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.069 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.058+0000 7f2b69a13e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.069 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.058+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.069 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.058+0000 7f2b69a13e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:18.088 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.078+0000 7f2b69a13e40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:00:18.088 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.078+0000 7f2b69a13e40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:00:18.088 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.078+0000 7f2b69a13e40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:00:18.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.078+0000 7f2b69a13e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:00:18.089 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.078+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.090 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.078+0000 7f2b69a13e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:18.091 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.078+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.091 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.078+0000 7f2b69a13e40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:00:18.092 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.078+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.092 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:18.092 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.092 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:00:18.092 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.093 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.094 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 20 zone default found 2023-01-24T17:00:18.094 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 4 Realm: () 2023-01-24T17:00:18.094 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 4 ZoneGroup: default (0d4b5768-74c8-4719-bc1b-4550aa389781) 2023-01-24T17:00:18.094 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 4 Zone: default (2ee3967b-537d-4521-a85e-a66f0a1f41f2) 2023-01-24T17:00:18.094 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:00:18.095 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 20 zonegroup default 2023-01-24T17:00:18.095 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.095 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.095 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.096 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.096 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.096 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.096 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 20 started sync module instance, tier type = 2023-01-24T17:00:18.096 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.082+0000 7f2b69a13e40 20 started zone id=2ee3967b-537d-4521-a85e-a66f0a1f41f2 (name=default) with tier type = 2023-01-24T17:00:18.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.102+0000 7f2b69a13e40 20 add_watcher() i=0 2023-01-24T17:00:18.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.102+0000 7f2b69a13e40 20 add_watcher() i=1 2023-01-24T17:00:18.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.102+0000 7f2b69a13e40 20 add_watcher() i=2 2023-01-24T17:00:18.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.102+0000 7f2b69a13e40 20 add_watcher() i=3 2023-01-24T17:00:18.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.102+0000 7f2b69a13e40 20 add_watcher() i=4 2023-01-24T17:00:18.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.102+0000 7f2b69a13e40 20 add_watcher() i=5 2023-01-24T17:00:18.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.102+0000 7f2b69a13e40 20 add_watcher() i=6 2023-01-24T17:00:18.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.106+0000 7f2b69a13e40 20 add_watcher() i=7 2023-01-24T17:00:18.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.106+0000 7f2b69a13e40 2 all 8 watchers are set, enabling cache 2023-01-24T17:00:18.116 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.106+0000 7f2ab3fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:00:18.116 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.106+0000 7f2b69a13e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:00:18.116 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.106+0000 7f2b69a13e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:00:18.123 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.114+0000 7f2b69a13e40 5 note: GC not initialized 2023-01-24T17:00:18.125 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.114+0000 7f2aaaffd700 20 reqs_thread_entry: start 2023-01-24T17:00:18.126 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2b69a13e40 20 init_complete bucket index max shards: 11 2023-01-24T17:00:18.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2aa8ff9700 20 reqs_thread_entry: start 2023-01-24T17:00:18.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2b69a13e40 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:00:18.127 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2a96ffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:00:48 2023 2023-01-24T17:00:18.127 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-01-24T17:00:18.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2a967fc700 20 reqs_thread_entry: start 2023-01-24T17:00:18.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2b69a13e40 20 get_system_obj_state: rctx=0x7fffab4c4148 obj=default.rgw.meta:users.uid:bar.client.0 state=0x562499b1def0 s->prefetch_data=0 2023-01-24T17:00:18.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2b69a13e40 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-01-24T17:00:18.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2b69a13e40 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-01-24T17:00:18.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2b69a13e40 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:00:18.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2b69a13e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:00:18.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2b69a13e40 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:00:18.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2b69a13e40 20 Read xattr: user.rgw.idtag 2023-01-24T17:00:18.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2b69a13e40 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:00:18.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2b69a13e40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2b69a13e40 20 rados_obj.operate() r=0 bl.length=455 2023-01-24T17:00:18.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2b69a13e40 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-01-24T17:00:18.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2b69a13e40 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:00:18.132 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.118+0000 7f2b69a13e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:00:18.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.122+0000 7f2b69a13e40 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-01-24T17:00:18.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.122+0000 7f2b69a13e40 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:00:18.133 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.122+0000 7f2b69a13e40 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-01-24T17:00:18.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.126+0000 7f2b6554a700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018883 cookie 94715198849744 notifier 4229 bl.length()=654 2023-01-24T17:00:18.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.126+0000 7f2b6554a700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2023-01-24T17:00:18.134 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.126+0000 7f2b6554a700 10 rgw watcher librados: moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:00:18.136 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.126+0000 7f2b69a13e40 20 remove_watcher() i=0 2023-01-24T17:00:18.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.126+0000 7f2b69a13e40 2 removed watcher, disabling cache 2023-01-24T17:00:18.139 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.130+0000 7f2b69a13e40 20 remove_watcher() i=1 2023-01-24T17:00:18.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.134+0000 7f2b69a13e40 20 remove_watcher() i=2 2023-01-24T17:00:18.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.138+0000 7f2b69a13e40 20 remove_watcher() i=3 2023-01-24T17:00:18.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.138+0000 7f2b69a13e40 20 remove_watcher() i=4 2023-01-24T17:00:18.151 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.142+0000 7f2b69a13e40 20 remove_watcher() i=5 2023-01-24T17:00:18.153 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.146+0000 7f2b69a13e40 20 remove_watcher() i=6 2023-01-24T17:00:18.155 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.146+0000 7f2b69a13e40 20 remove_watcher() i=7 2023-01-24T17:00:18.157 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.146+0000 7f2ab3fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:00:18.167 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2023-01-24T17:00:18.168 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --access-key FQIERVIEZQHTHXREXVKU --secret MdTTIy80K6OeaE3o0LTy+nHuenrtn5ylbN60kk17+7Omwxn67jJepQ== --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --cluster ceph 2023-01-24T17:00:18.214 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:00:18.214 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:00:18.226 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.218+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.218+0000 7fdc66afae40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.218+0000 7fdc66afae40 20 realm 2023-01-24T17:00:18.229 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.218+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.230 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.222+0000 7fdc66afae40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.230 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.222+0000 7fdc66afae40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:00:18.230 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.222+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.231 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.222+0000 7fdc66afae40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.231 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.222+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.231 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.222+0000 7fdc66afae40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:18.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.222+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.222+0000 7fdc66afae40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:00:18.232 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.222+0000 7fdc66afae40 20 searching for the correct realm 2023-01-24T17:00:18.253 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.242+0000 7fdc66afae40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:00:18.253 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.242+0000 7fdc66afae40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:00:18.254 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.242+0000 7fdc66afae40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:00:18.254 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.242+0000 7fdc66afae40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:00:18.254 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.242+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.246+0000 7fdc66afae40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.246+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.246+0000 7fdc66afae40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:18.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.246+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.246+0000 7fdc66afae40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:00:18.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.246+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.246+0000 7fdc66afae40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.246+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.246+0000 7fdc66afae40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.246+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.258 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.246+0000 7fdc66afae40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:18.280 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.270+0000 7fdc66afae40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:00:18.281 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.270+0000 7fdc66afae40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:00:18.281 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.270+0000 7fdc66afae40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:00:18.281 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.270+0000 7fdc66afae40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:00:18.281 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.270+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.282 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.270+0000 7fdc66afae40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:18.282 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.270+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.283 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.274+0000 7fdc66afae40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:00:18.283 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.274+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.283 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.274+0000 7fdc66afae40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:18.284 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.274+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.284 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.274+0000 7fdc66afae40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:00:18.284 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.274+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.274+0000 7fdc66afae40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.274+0000 7fdc66afae40 20 zone default found 2023-01-24T17:00:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.274+0000 7fdc66afae40 4 Realm: () 2023-01-24T17:00:18.286 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.274+0000 7fdc66afae40 4 ZoneGroup: default (0d4b5768-74c8-4719-bc1b-4550aa389781) 2023-01-24T17:00:18.286 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.274+0000 7fdc66afae40 4 Zone: default (2ee3967b-537d-4521-a85e-a66f0a1f41f2) 2023-01-24T17:00:18.286 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.274+0000 7fdc66afae40 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:00:18.286 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.274+0000 7fdc66afae40 20 zonegroup default 2023-01-24T17:00:18.286 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.274+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.287 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.274+0000 7fdc66afae40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.287 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.274+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.288 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.278+0000 7fdc66afae40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.289 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.278+0000 7fdc66afae40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.289 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.278+0000 7fdc66afae40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.289 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.278+0000 7fdc66afae40 20 started sync module instance, tier type = 2023-01-24T17:00:18.289 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.278+0000 7fdc66afae40 20 started zone id=2ee3967b-537d-4521-a85e-a66f0a1f41f2 (name=default) with tier type = 2023-01-24T17:00:18.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.298+0000 7fdc66afae40 20 add_watcher() i=0 2023-01-24T17:00:18.308 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.298+0000 7fdc66afae40 20 add_watcher() i=1 2023-01-24T17:00:18.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.298+0000 7fdc66afae40 20 add_watcher() i=2 2023-01-24T17:00:18.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.298+0000 7fdc66afae40 20 add_watcher() i=3 2023-01-24T17:00:18.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.298+0000 7fdc66afae40 20 add_watcher() i=4 2023-01-24T17:00:18.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.298+0000 7fdc66afae40 20 add_watcher() i=5 2023-01-24T17:00:18.309 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.298+0000 7fdc66afae40 20 add_watcher() i=6 2023-01-24T17:00:18.311 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.302+0000 7fdc66afae40 20 add_watcher() i=7 2023-01-24T17:00:18.311 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.302+0000 7fdc66afae40 2 all 8 watchers are set, enabling cache 2023-01-24T17:00:18.312 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.302+0000 7fdc35fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:00:18.312 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.302+0000 7fdc66afae40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:00:18.312 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.302+0000 7fdc66afae40 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:00:18.318 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.310+0000 7fdc66afae40 5 note: GC not initialized 2023-01-24T17:00:18.320 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.310+0000 7fdbacff9700 20 reqs_thread_entry: start 2023-01-24T17:00:18.320 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.310+0000 7fdc66afae40 20 init_complete bucket index max shards: 11 2023-01-24T17:00:18.321 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.310+0000 7fdc66afae40 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:00:18.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.310+0000 7fdba0ff9700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:00:48 2023 2023-01-24T17:00:18.322 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-01-24T17:00:18.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.310+0000 7fdb8ffff700 20 reqs_thread_entry: start 2023-01-24T17:00:18.322 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.310+0000 7fdc66afae40 20 get_system_obj_state: rctx=0x7ffcc0c5a578 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x563a0fd89ef0 s->prefetch_data=0 2023-01-24T17:00:18.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.310+0000 7fdc66afae40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-01-24T17:00:18.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.310+0000 7fdba27fc700 20 reqs_thread_entry: start 2023-01-24T17:00:18.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.310+0000 7fdc66afae40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2023-01-24T17:00:18.323 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc66afae40 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:00:18.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc66afae40 20 get_system_obj_state: rctx=0x7ffcc0c5a268 obj=default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test state=0x563a0fd89ef0 s->prefetch_data=0 2023-01-24T17:00:18.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc66afae40 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2023-01-24T17:00:18.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc66afae40 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2023-01-24T17:00:18.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc66afae40 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-01-24T17:00:18.324 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc66afae40 20 get_system_obj_state: rctx=0x7ffcc0c5a268 obj=default.rgw.meta:users.keys:FQIERVIEZQHTHXREXVKU state=0x563a0fd89ef0 s->prefetch_data=0 2023-01-24T17:00:18.325 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc66afae40 10 cache get: name=default.rgw.meta+users.keys+FQIERVIEZQHTHXREXVKU : miss 2023-01-24T17:00:18.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc66afae40 10 cache put: name=default.rgw.meta+users.keys+FQIERVIEZQHTHXREXVKU info.flags=0x0 2023-01-24T17:00:18.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc66afae40 10 adding default.rgw.meta+users.keys+FQIERVIEZQHTHXREXVKU to cache LRU end 2023-01-24T17:00:18.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc66afae40 20 get_system_obj_state: rctx=0x7ffcc0c59b08 obj=default.rgw.meta:users.keys:FQIERVIEZQHTHXREXVKU state=0x563a0fd89ef0 s->prefetch_data=0 2023-01-24T17:00:18.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc66afae40 10 cache get: name=default.rgw.meta+users.keys+FQIERVIEZQHTHXREXVKU : hit (negative entry) 2023-01-24T17:00:18.326 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc66afae40 20 get_system_obj_state: rctx=0x7ffcc0c5a1a8 obj=default.rgw.meta:users.keys:FQIERVIEZQHTHXREXVKU state=0x563a0fd89ef0 s->prefetch_data=0 2023-01-24T17:00:18.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc66afae40 10 cache get: name=default.rgw.meta+users.keys+FQIERVIEZQHTHXREXVKU : hit (negative entry) 2023-01-24T17:00:18.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc66afae40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-01-24T17:00:18.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc66afae40 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:00:18.327 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc66afae40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-01-24T17:00:18.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc62631700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018883 cookie 94807375422944 notifier 4235 bl.length()=714 2023-01-24T17:00:18.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc62631700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-01-24T17:00:18.328 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.314+0000 7fdc62631700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:00:18.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.318+0000 7fdc66afae40 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-01-24T17:00:18.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.318+0000 7fdc66afae40 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-01-24T17:00:18.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.318+0000 7fdc66afae40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-01-24T17:00:18.329 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.318+0000 7fdc61e30700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018884 cookie 94807375429648 notifier 4235 bl.length()=224 2023-01-24T17:00:18.330 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.318+0000 7fdc61e30700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2023-01-24T17:00:18.330 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.318+0000 7fdc61e30700 10 rgw watcher librados: moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2023-01-24T17:00:18.331 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.322+0000 7fdc66afae40 10 cache put: name=default.rgw.meta+users.keys+FQIERVIEZQHTHXREXVKU info.flags=0x7 2023-01-24T17:00:18.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.322+0000 7fdc66afae40 10 moving default.rgw.meta+users.keys+FQIERVIEZQHTHXREXVKU to cache LRU end 2023-01-24T17:00:18.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.322+0000 7fdc66afae40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.keys:FQIERVIEZQHTHXREXVKU, ofs0, ns] 2023-01-24T17:00:18.332 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.322+0000 7fdc62631700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018884 cookie 94807375432000 notifier 4235 bl.length()=201 2023-01-24T17:00:18.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.322+0000 7fdc62631700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+FQIERVIEZQHTHXREXVKU info.flags=0x7 2023-01-24T17:00:18.333 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.322+0000 7fdc62631700 10 rgw watcher librados: moving default.rgw.meta+users.keys+FQIERVIEZQHTHXREXVKU to cache LRU end 2023-01-24T17:00:18.334 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2023-01-24T17:00:18.334 INFO:teuthology.orchestra.run.smithi103.stdout: "user_id": "testx$tenanteduser.client.0", 2023-01-24T17:00:18.334 INFO:teuthology.orchestra.run.smithi103.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2023-01-24T17:00:18.334 INFO:teuthology.orchestra.run.smithi103.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2023-01-24T17:00:18.334 INFO:teuthology.orchestra.run.smithi103.stdout: "suspended": 0, 2023-01-24T17:00:18.335 INFO:teuthology.orchestra.run.smithi103.stdout: "max_buckets": 1000, 2023-01-24T17:00:18.335 INFO:teuthology.orchestra.run.smithi103.stdout: "subusers": [], 2023-01-24T17:00:18.335 INFO:teuthology.orchestra.run.smithi103.stdout: "keys": [ 2023-01-24T17:00:18.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2023-01-24T17:00:18.336 INFO:teuthology.orchestra.run.smithi103.stdout: "user": "testx$tenanteduser.client.0", 2023-01-24T17:00:18.337 INFO:teuthology.orchestra.run.smithi103.stdout: "access_key": "FQIERVIEZQHTHXREXVKU", 2023-01-24T17:00:18.337 INFO:teuthology.orchestra.run.smithi103.stdout: "secret_key": "MdTTIy80K6OeaE3o0LTy+nHuenrtn5ylbN60kk17+7Omwxn67jJepQ==" 2023-01-24T17:00:18.337 INFO:teuthology.orchestra.run.smithi103.stdout: } 2023-01-24T17:00:18.337 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2023-01-24T17:00:18.338 INFO:teuthology.orchestra.run.smithi103.stdout: "swift_keys": [], 2023-01-24T17:00:18.338 INFO:teuthology.orchestra.run.smithi103.stdout: "caps": [ 2023-01-24T17:00:18.338 INFO:teuthology.orchestra.run.smithi103.stdout: { 2023-01-24T17:00:18.338 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "user-policy", 2023-01-24T17:00:18.338 INFO:teuthology.orchestra.run.smithi103.stdout: "perm": "*" 2023-01-24T17:00:18.339 INFO:teuthology.orchestra.run.smithi103.stdout: } 2023-01-24T17:00:18.339 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2023-01-24T17:00:18.339 INFO:teuthology.orchestra.run.smithi103.stdout: "op_mask": "read, write, delete", 2023-01-24T17:00:18.339 INFO:teuthology.orchestra.run.smithi103.stdout: "default_placement": "", 2023-01-24T17:00:18.339 INFO:teuthology.orchestra.run.smithi103.stdout: "default_storage_class": "", 2023-01-24T17:00:18.339 INFO:teuthology.orchestra.run.smithi103.stdout: "placement_tags": [], 2023-01-24T17:00:18.340 INFO:teuthology.orchestra.run.smithi103.stdout: "bucket_quota": { 2023-01-24T17:00:18.340 INFO:teuthology.orchestra.run.smithi103.stdout: "enabled": false, 2023-01-24T17:00:18.340 INFO:teuthology.orchestra.run.smithi103.stdout: "check_on_raw": false, 2023-01-24T17:00:18.340 INFO:teuthology.orchestra.run.smithi103.stdout: "max_size": -1, 2023-01-24T17:00:18.340 INFO:teuthology.orchestra.run.smithi103.stdout: "max_size_kb": 0, 2023-01-24T17:00:18.341 INFO:teuthology.orchestra.run.smithi103.stdout: "max_objects": -1 2023-01-24T17:00:18.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2023-01-24T17:00:18.341 INFO:teuthology.orchestra.run.smithi103.stdout: "user_quota": { 2023-01-24T17:00:18.341 INFO:teuthology.orchestra.run.smithi103.stdout: "enabled": false, 2023-01-24T17:00:18.341 INFO:teuthology.orchestra.run.smithi103.stdout: "check_on_raw": false, 2023-01-24T17:00:18.342 INFO:teuthology.orchestra.run.smithi103.stdout: "max_size": -1, 2023-01-24T17:00:18.342 INFO:teuthology.orchestra.run.smithi103.stdout: "max_size_kb": 0, 2023-01-24T17:00:18.343 INFO:teuthology.orchestra.run.smithi103.stdout: "max_objects": -1 2023-01-24T17:00:18.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2023-01-24T17:00:18.343 INFO:teuthology.orchestra.run.smithi103.stdout: "temp_url_keys": [], 2023-01-24T17:00:18.343 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "rgw", 2023-01-24T17:00:18.343 INFO:teuthology.orchestra.run.smithi103.stdout: "mfa_ids": [] 2023-01-24T17:00:18.344 INFO:teuthology.orchestra.run.smithi103.stdout:} 2023-01-24T17:00:18.344 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:18.345 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.326+0000 7fdc66afae40 20 remove_watcher() i=0 2023-01-24T17:00:18.345 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.326+0000 7fdc66afae40 2 removed watcher, disabling cache 2023-01-24T17:00:18.345 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.330+0000 7fdc66afae40 20 remove_watcher() i=1 2023-01-24T17:00:18.346 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.330+0000 7fdc66afae40 20 remove_watcher() i=2 2023-01-24T17:00:18.346 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.334+0000 7fdc66afae40 20 remove_watcher() i=3 2023-01-24T17:00:18.346 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.338+0000 7fdc66afae40 20 remove_watcher() i=4 2023-01-24T17:00:18.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.338+0000 7fdc66afae40 20 remove_watcher() i=5 2023-01-24T17:00:18.350 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.342+0000 7fdc66afae40 20 remove_watcher() i=6 2023-01-24T17:00:18.353 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.342+0000 7fdc66afae40 20 remove_watcher() i=7 2023-01-24T17:00:18.354 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.346+0000 7fdc35fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:00:18.364 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 9704860285 --totp-seed CSHCVJVBU7D6Q2673SZ3VRZKJUA2ZAGSQR2YIXEC5B4I4WTAIM3XIEI22XZADM6V --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2023-01-24T17:00:18.407 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:00:18.407 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:00:18.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.410+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.410+0000 7fa3236fde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.410+0000 7fa3236fde40 20 realm 2023-01-24T17:00:18.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.410+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.421 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.410+0000 7fa3236fde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.410+0000 7fa3236fde40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:00:18.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.410+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.414+0000 7fa3236fde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.414+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.414+0000 7fa3236fde40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:18.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.414+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.414+0000 7fa3236fde40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:00:18.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.414+0000 7fa3236fde40 20 searching for the correct realm 2023-01-24T17:00:18.445 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.434+0000 7fa3236fde40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:00:18.445 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.434+0000 7fa3236fde40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:00:18.445 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.434+0000 7fa3236fde40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:00:18.446 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.434+0000 7fa3236fde40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:00:18.446 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.434+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.447 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.434+0000 7fa3236fde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.447 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.434+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.447 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.438+0000 7fa3236fde40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:18.447 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.438+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.447 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.438+0000 7fa3236fde40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:00:18.448 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.438+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.448 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.438+0000 7fa3236fde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.448 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.438+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.449 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.438+0000 7fa3236fde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.449 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.438+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.449 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.438+0000 7fa3236fde40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:18.468 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.458+0000 7fa3236fde40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:00:18.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.458+0000 7fa3236fde40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:00:18.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.458+0000 7fa3236fde40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:00:18.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.458+0000 7fa3236fde40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:00:18.469 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.458+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.458+0000 7fa3236fde40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:18.470 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.458+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.458+0000 7fa3236fde40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:00:18.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:00:18.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.471 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:00:18.472 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 20 zone default found 2023-01-24T17:00:18.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 4 Realm: () 2023-01-24T17:00:18.473 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 4 ZoneGroup: default (0d4b5768-74c8-4719-bc1b-4550aa389781) 2023-01-24T17:00:18.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 4 Zone: default (2ee3967b-537d-4521-a85e-a66f0a1f41f2) 2023-01-24T17:00:18.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:00:18.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 20 zonegroup default 2023-01-24T17:00:18.474 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.475 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:00:18.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 20 started sync module instance, tier type = 2023-01-24T17:00:18.476 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.462+0000 7fa3236fde40 20 started zone id=2ee3967b-537d-4521-a85e-a66f0a1f41f2 (name=default) with tier type = 2023-01-24T17:00:18.501 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.490+0000 7fa3236fde40 20 add_watcher() i=0 2023-01-24T17:00:18.501 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.490+0000 7fa3236fde40 20 add_watcher() i=1 2023-01-24T17:00:18.501 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.490+0000 7fa3236fde40 20 add_watcher() i=2 2023-01-24T17:00:18.502 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.490+0000 7fa3236fde40 20 add_watcher() i=3 2023-01-24T17:00:18.502 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.490+0000 7fa3236fde40 20 add_watcher() i=4 2023-01-24T17:00:18.502 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.490+0000 7fa3236fde40 20 add_watcher() i=5 2023-01-24T17:00:18.503 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.490+0000 7fa3236fde40 20 add_watcher() i=6 2023-01-24T17:00:18.503 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.494+0000 7fa3236fde40 20 add_watcher() i=7 2023-01-24T17:00:18.503 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.494+0000 7fa3236fde40 2 all 8 watchers are set, enabling cache 2023-01-24T17:00:18.505 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.498+0000 7fa3236fde40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:00:18.506 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.498+0000 7fa3236fde40 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:00:18.506 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.498+0000 7fa2f1feb700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:00:18.513 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.502+0000 7fa3236fde40 5 note: GC not initialized 2023-01-24T17:00:18.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.506+0000 7fa268ff9700 20 reqs_thread_entry: start 2023-01-24T17:00:18.516 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.506+0000 7fa3236fde40 20 init_complete bucket index max shards: 11 2023-01-24T17:00:18.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.506+0000 7fa25e7fc700 20 reqs_thread_entry: start 2023-01-24T17:00:18.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.506+0000 7fa3236fde40 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:00:18.517 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.506+0000 7fa25cff9700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:00:48 2023 2023-01-24T17:00:18.518 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-01-24T17:00:18.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.506+0000 7fa3236fde40 20 get_system_obj_state: rctx=0x7ffe83443188 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x55c424d1aef0 s->prefetch_data=0 2023-01-24T17:00:18.518 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.506+0000 7fa3236fde40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-01-24T17:00:18.519 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.506+0000 7fa243fff700 20 reqs_thread_entry: start 2023-01-24T17:00:18.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.506+0000 7fa3236fde40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-01-24T17:00:18.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.506+0000 7fa3236fde40 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:00:18.520 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.506+0000 7fa3236fde40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:00:18.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.506+0000 7fa3236fde40 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:00:18.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.506+0000 7fa3236fde40 20 Read xattr: user.rgw.idtag 2023-01-24T17:00:18.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.506+0000 7fa3236fde40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:00:18.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.506+0000 7fa3236fde40 20 rados->read ofs=0 len=0 2023-01-24T17:00:18.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.510+0000 7fa3236fde40 20 rados_obj.operate() r=0 bl.length=514 2023-01-24T17:00:18.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.510+0000 7fa3236fde40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-01-24T17:00:18.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.510+0000 7fa3236fde40 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:00:18.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.510+0000 7fa3236fde40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:00:18.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.514+0000 7fa3236fde40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-01-24T17:00:18.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.514+0000 7fa3236fde40 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:00:18.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.514+0000 7fa3236fde40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-01-24T17:00:18.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.514+0000 7fa31affd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 128849018885 cookie 94300921144352 notifier 4241 bl.length()=728 2023-01-24T17:00:18.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.514+0000 7fa31affd700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2023-01-24T17:00:18.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.514+0000 7fa31affd700 10 rgw watcher librados: moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:00:18.528 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.518+0000 7fa3236fde40 20 remove_watcher() i=0 2023-01-24T17:00:18.528 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.518+0000 7fa3236fde40 2 removed watcher, disabling cache 2023-01-24T17:00:18.530 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.522+0000 7fa3236fde40 20 remove_watcher() i=1 2023-01-24T17:00:18.532 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.522+0000 7fa3236fde40 20 remove_watcher() i=2 2023-01-24T17:00:18.534 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.526+0000 7fa3236fde40 20 remove_watcher() i=3 2023-01-24T17:00:18.536 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.526+0000 7fa3236fde40 20 remove_watcher() i=4 2023-01-24T17:00:18.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.530+0000 7fa3236fde40 20 remove_watcher() i=5 2023-01-24T17:00:18.540 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.530+0000 7fa3236fde40 20 remove_watcher() i=6 2023-01-24T17:00:18.542 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.534+0000 7fa3236fde40 20 remove_watcher() i=7 2023-01-24T17:00:18.544 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:00:18.534+0000 7fa2f1feb700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:00:18.554 INFO:tasks.s3tests:Configuring s3-tests... 2023-01-24T17:00:18.554 DEBUG:teuthology.orchestra.run.smithi103:> cd /home/ubuntu/cephtest/s3-tests && ./bootstrap 2023-01-24T17:00:18.735 INFO:teuthology.orchestra.run.smithi103.stderr:./bootstrap: missing required DEB packages. Installing via sudo. 2023-01-24T17:00:18.797 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2023-01-24T17:00:18.987 INFO:teuthology.orchestra.run.smithi103.stdout:Building dependency tree... 2023-01-24T17:00:18.988 INFO:teuthology.orchestra.run.smithi103.stdout:Reading state information... 2023-01-24T17:00:19.159 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages were automatically installed and are no longer required: 2023-01-24T17:00:19.160 INFO:teuthology.orchestra.run.smithi103.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-01-24T17:00:19.160 INFO:teuthology.orchestra.run.smithi103.stdout:Use 'sudo apt autoremove' to remove them. 2023-01-24T17:00:19.161 INFO:teuthology.orchestra.run.smithi103.stdout:The following additional packages will be installed: 2023-01-24T17:00:19.161 INFO:teuthology.orchestra.run.smithi103.stdout: libevent-extra-2.1-7 libevent-openssl-2.1-7 libxml2 libxslt1.1 2023-01-24T17:00:19.162 INFO:teuthology.orchestra.run.smithi103.stdout:Suggested packages: 2023-01-24T17:00:19.162 INFO:teuthology.orchestra.run.smithi103.stdout: pkg-config 2023-01-24T17:00:19.197 INFO:teuthology.orchestra.run.smithi103.stdout:The following NEW packages will be installed: 2023-01-24T17:00:19.197 INFO:teuthology.orchestra.run.smithi103.stdout: libevent-dev libevent-extra-2.1-7 libevent-openssl-2.1-7 libxml2-dev 2023-01-24T17:00:19.198 INFO:teuthology.orchestra.run.smithi103.stdout: libxslt1-dev 2023-01-24T17:00:19.199 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages will be upgraded: 2023-01-24T17:00:19.200 INFO:teuthology.orchestra.run.smithi103.stdout: libxml2 libxslt1.1 2023-01-24T17:00:19.381 INFO:teuthology.orchestra.run.smithi103.stdout:2 upgraded, 5 newly installed, 0 to remove and 117 not upgraded. 2023-01-24T17:00:19.381 INFO:teuthology.orchestra.run.smithi103.stdout:Need to get 2,081 kB of archives. 2023-01-24T17:00:19.381 INFO:teuthology.orchestra.run.smithi103.stdout:After this operation, 7,578 kB of additional disk space will be used. 2023-01-24T17:00:19.382 INFO:teuthology.orchestra.run.smithi103.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libxml2 amd64 2.9.10+dfsg-5ubuntu0.20.04.5 [640 kB] 2023-01-24T17:00:19.938 INFO:teuthology.orchestra.run.smithi103.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libevent-extra-2.1-7 amd64 2.1.11-stable-1 [60.0 kB] 2023-01-24T17:00:19.942 INFO:teuthology.orchestra.run.smithi103.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libevent-openssl-2.1-7 amd64 2.1.11-stable-1 [14.3 kB] 2023-01-24T17:00:19.943 INFO:teuthology.orchestra.run.smithi103.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libevent-dev amd64 2.1.11-stable-1 [261 kB] 2023-01-24T17:00:19.968 INFO:teuthology.orchestra.run.smithi103.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libxml2-dev amd64 2.9.10+dfsg-5ubuntu0.20.04.5 [735 kB] 2023-01-24T17:00:20.055 INFO:teuthology.orchestra.run.smithi103.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libxslt1.1 amd64 1.1.34-4ubuntu0.20.04.1 [151 kB] 2023-01-24T17:00:20.068 INFO:teuthology.orchestra.run.smithi103.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libxslt1-dev amd64 1.1.34-4ubuntu0.20.04.1 [219 kB] 2023-01-24T17:00:20.408 INFO:teuthology.orchestra.run.smithi103.stderr:debconf: unable to initialize frontend: Dialog 2023-01-24T17:00:20.408 INFO:teuthology.orchestra.run.smithi103.stderr:debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.) 2023-01-24T17:00:20.409 INFO:teuthology.orchestra.run.smithi103.stderr:debconf: falling back to frontend: Readline 2023-01-24T17:00:20.428 INFO:teuthology.orchestra.run.smithi103.stderr:debconf: unable to initialize frontend: Readline 2023-01-24T17:00:20.429 INFO:teuthology.orchestra.run.smithi103.stderr:debconf: (This frontend requires a controlling tty.) 2023-01-24T17:00:20.429 INFO:teuthology.orchestra.run.smithi103.stderr:debconf: falling back to frontend: Teletype 2023-01-24T17:00:20.436 INFO:teuthology.orchestra.run.smithi103.stderr:dpkg-preconfigure: unable to re-open stdin: 2023-01-24T17:00:20.493 INFO:teuthology.orchestra.run.smithi103.stdout:Fetched 2,081 kB in 1s (2,186 kB/s) 2023-01-24T17:00:20.694 INFO:teuthology.orchestra.run.smithi103.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152767 files and directories currently installed.) 2023-01-24T17:00:20.699 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../0-libxml2_2.9.10+dfsg-5ubuntu0.20.04.5_amd64.deb ... 2023-01-24T17:00:20.890 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libxml2:amd64 (2.9.10+dfsg-5ubuntu0.20.04.5) over (2.9.10+dfsg-5ubuntu0.20.04.4) ... 2023-01-24T17:00:21.354 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libevent-extra-2.1-7:amd64. 2023-01-24T17:00:21.375 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../1-libevent-extra-2.1-7_2.1.11-stable-1_amd64.deb ... 2023-01-24T17:00:21.412 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libevent-extra-2.1-7:amd64 (2.1.11-stable-1) ... 2023-01-24T17:00:21.759 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libevent-openssl-2.1-7:amd64. 2023-01-24T17:00:21.780 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../2-libevent-openssl-2.1-7_2.1.11-stable-1_amd64.deb ... 2023-01-24T17:00:21.826 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libevent-openssl-2.1-7:amd64 (2.1.11-stable-1) ... 2023-01-24T17:00:22.079 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libevent-dev. 2023-01-24T17:00:22.097 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../3-libevent-dev_2.1.11-stable-1_amd64.deb ... 2023-01-24T17:00:22.137 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libevent-dev (2.1.11-stable-1) ... 2023-01-24T17:00:22.486 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libxml2-dev:amd64. 2023-01-24T17:00:22.504 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../4-libxml2-dev_2.9.10+dfsg-5ubuntu0.20.04.5_amd64.deb ... 2023-01-24T17:00:22.544 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libxml2-dev:amd64 (2.9.10+dfsg-5ubuntu0.20.04.5) ... 2023-01-24T17:00:23.049 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../5-libxslt1.1_1.1.34-4ubuntu0.20.04.1_amd64.deb ... 2023-01-24T17:00:23.207 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4ubuntu0.20.04.1) over (1.1.34-4) ... 2023-01-24T17:00:23.547 INFO:teuthology.orchestra.run.smithi103.stdout:Selecting previously unselected package libxslt1-dev:amd64. 2023-01-24T17:00:23.565 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing to unpack .../6-libxslt1-dev_1.1.34-4ubuntu0.20.04.1_amd64.deb ... 2023-01-24T17:00:23.613 INFO:teuthology.orchestra.run.smithi103.stdout:Unpacking libxslt1-dev:amd64 (1.1.34-4ubuntu0.20.04.1) ... 2023-01-24T17:00:23.948 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libevent-extra-2.1-7:amd64 (2.1.11-stable-1) ... 2023-01-24T17:00:24.095 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libevent-openssl-2.1-7:amd64 (2.1.11-stable-1) ... 2023-01-24T17:00:24.212 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libevent-dev (2.1.11-stable-1) ... 2023-01-24T17:00:24.538 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libxml2:amd64 (2.9.10+dfsg-5ubuntu0.20.04.5) ... 2023-01-24T17:00:24.664 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libxml2-dev:amd64 (2.9.10+dfsg-5ubuntu0.20.04.5) ... 2023-01-24T17:00:24.823 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libxslt1.1:amd64 (1.1.34-4ubuntu0.20.04.1) ... 2023-01-24T17:00:24.949 INFO:teuthology.orchestra.run.smithi103.stdout:Setting up libxslt1-dev:amd64 (1.1.34-4ubuntu0.20.04.1) ... 2023-01-24T17:00:25.076 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-01-24T17:00:26.704 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-01-24T17:00:27.725 INFO:teuthology.orchestra.run.smithi103.stdout:created virtual environment CPython3.8.10.final.0-64 in 302ms 2023-01-24T17:00:27.725 INFO:teuthology.orchestra.run.smithi103.stdout: creator CPython3Posix(dest=/home/ubuntu/cephtest/s3-tests/virtualenv, clear=False, global=False) 2023-01-24T17:00:27.725 INFO:teuthology.orchestra.run.smithi103.stdout: seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, pkg_resources=latest, via=copy, app_data_dir=/home/ubuntu/.local/share/virtualenv/seed-app-data/v1.0.1.debian.1) 2023-01-24T17:00:27.725 INFO:teuthology.orchestra.run.smithi103.stdout: activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator 2023-01-24T17:00:28.962 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting pip 2023-01-24T17:00:28.968 INFO:teuthology.orchestra.run.smithi103.stdout: Using cached pip-22.3.1-py3-none-any.whl (2.1 MB) 2023-01-24T17:00:29.067 INFO:teuthology.orchestra.run.smithi103.stdout:Installing collected packages: pip 2023-01-24T17:00:29.067 INFO:teuthology.orchestra.run.smithi103.stdout: Attempting uninstall: pip 2023-01-24T17:00:29.067 INFO:teuthology.orchestra.run.smithi103.stdout: Found existing installation: pip 20.0.2 2023-01-24T17:00:29.115 INFO:teuthology.orchestra.run.smithi103.stdout: Uninstalling pip-20.0.2: 2023-01-24T17:00:29.118 INFO:teuthology.orchestra.run.smithi103.stdout: Successfully uninstalled pip-20.0.2 2023-01-24T17:00:29.988 INFO:teuthology.orchestra.run.smithi103.stdout:Successfully installed pip-22.3.1 2023-01-24T17:00:30.738 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting setuptools==44.1.0 2023-01-24T17:00:30.839 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB) 2023-01-24T17:00:30.932 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 583.4/583.4 kB 6.6 MB/s eta 0:00:00 2023-01-24T17:00:30.949 INFO:teuthology.orchestra.run.smithi103.stdout:Installing collected packages: setuptools 2023-01-24T17:00:30.950 INFO:teuthology.orchestra.run.smithi103.stdout: Attempting uninstall: setuptools 2023-01-24T17:00:30.950 INFO:teuthology.orchestra.run.smithi103.stdout: Found existing installation: setuptools 44.0.0 2023-01-24T17:00:30.974 INFO:teuthology.orchestra.run.smithi103.stdout: Uninstalling setuptools-44.0.0: 2023-01-24T17:00:30.980 INFO:teuthology.orchestra.run.smithi103.stdout: Successfully uninstalled setuptools-44.0.0 2023-01-24T17:00:31.230 INFO:teuthology.orchestra.run.smithi103.stdout:Successfully installed setuptools-44.1.0 2023-01-24T17:00:32.043 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting PyYAML 2023-01-24T17:00:32.140 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB) 2023-01-24T17:00:32.245 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 701.2/701.2 kB 7.0 MB/s eta 0:00:00 2023-01-24T17:00:32.315 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting nose>=1.0.0 2023-01-24T17:00:32.334 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading nose-1.3.7-py3-none-any.whl (154 kB) 2023-01-24T17:00:32.370 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 154.7/154.7 kB 4.7 MB/s eta 0:00:00 2023-01-24T17:00:32.458 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting boto>=2.6.0 2023-01-24T17:00:32.478 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2023-01-24T17:00:32.564 INFO:teuthology.orchestra.run.smithi103.stdout: ---------------------------------------- 1.4/1.4 MB 16.8 MB/s eta 0:00:00 2023-01-24T17:00:33.123 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting boto3>=1.0.0 2023-01-24T17:00:33.141 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.55-py3-none-any.whl (132 kB) 2023-01-24T17:00:33.179 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 3.6 MB/s eta 0:00:00 2023-01-24T17:00:33.771 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting botocore<1.28.0 2023-01-24T17:00:33.809 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading botocore-1.27.96-py3-none-any.whl (9.3 MB) 2023-01-24T17:00:34.186 INFO:teuthology.orchestra.run.smithi103.stdout: ---------------------------------------- 9.3/9.3 MB 24.9 MB/s eta 0:00:00 2023-01-24T17:00:34.269 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting munch>=2.0.0 2023-01-24T17:00:34.287 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading munch-2.5.0-py2.py3-none-any.whl (10 kB) 2023-01-24T17:00:34.595 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting gevent>=1.0 2023-01-24T17:00:34.615 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading gevent-22.10.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.5 MB) 2023-01-24T17:00:34.804 INFO:teuthology.orchestra.run.smithi103.stdout: ---------------------------------------- 6.5/6.5 MB 35.4 MB/s eta 0:00:00 2023-01-24T17:00:34.885 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting isodate>=0.4.4 2023-01-24T17:00:34.903 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB) 2023-01-24T17:00:34.934 INFO:teuthology.orchestra.run.smithi103.stdout: ---------------------------------------- 41.7/41.7 kB 1.2 MB/s eta 0:00:00 2023-01-24T17:00:35.030 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting requests>=2.23.0 2023-01-24T17:00:35.032 INFO:teuthology.orchestra.run.smithi103.stdout: Using cached requests-2.28.2-py3-none-any.whl (62 kB) 2023-01-24T17:00:35.481 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting pytz>=2011k 2023-01-24T17:00:35.482 INFO:teuthology.orchestra.run.smithi103.stdout: Using cached pytz-2022.7.1-py2.py3-none-any.whl (499 kB) 2023-01-24T17:00:35.597 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting httplib2 2023-01-24T17:00:35.640 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading httplib2-0.21.0-py3-none-any.whl (96 kB) 2023-01-24T17:00:35.679 INFO:teuthology.orchestra.run.smithi103.stdout: ---------------------------------------- 96.8/96.8 kB 2.6 MB/s eta 0:00:00 2023-01-24T17:00:36.127 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting lxml 2023-01-24T17:00:36.149 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading lxml-4.9.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (7.1 MB) 2023-01-24T17:00:36.449 INFO:teuthology.orchestra.run.smithi103.stdout: ---------------------------------------- 7.1/7.1 MB 24.1 MB/s eta 0:00:00 2023-01-24T17:00:36.563 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting jmespath<2.0.0,>=0.7.1 2023-01-24T17:00:36.580 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading jmespath-1.0.1-py3-none-any.whl (20 kB) 2023-01-24T17:00:36.751 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting boto3>=1.0.0 2023-01-24T17:00:36.769 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.54-py3-none-any.whl (132 kB) 2023-01-24T17:00:36.806 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 3.8 MB/s eta 0:00:00 2023-01-24T17:00:36.892 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting s3transfer<0.7.0,>=0.6.0 2023-01-24T17:00:36.910 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading s3transfer-0.6.0-py3-none-any.whl (79 kB) 2023-01-24T17:00:36.940 INFO:teuthology.orchestra.run.smithi103.stdout: ---------------------------------------- 79.6/79.6 kB 2.8 MB/s eta 0:00:00 2023-01-24T17:00:37.044 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting boto3>=1.0.0 2023-01-24T17:00:37.065 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.53-py3-none-any.whl (132 kB) 2023-01-24T17:00:37.108 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 3.2 MB/s eta 0:00:00 2023-01-24T17:00:37.228 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.52-py3-none-any.whl (132 kB) 2023-01-24T17:00:37.267 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 3.6 MB/s eta 0:00:00 2023-01-24T17:00:37.432 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.51-py3-none-any.whl (132 kB) 2023-01-24T17:00:37.466 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 4.0 MB/s eta 0:00:00 2023-01-24T17:00:37.582 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.50-py3-none-any.whl (132 kB) 2023-01-24T17:00:37.617 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 4.0 MB/s eta 0:00:00 2023-01-24T17:00:37.836 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.49-py3-none-any.whl (132 kB) 2023-01-24T17:00:37.876 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 3.5 MB/s eta 0:00:00 2023-01-24T17:00:38.080 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.48-py3-none-any.whl (132 kB) 2023-01-24T17:00:38.110 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 4.9 MB/s eta 0:00:00 2023-01-24T17:00:38.345 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.47-py3-none-any.whl (132 kB) 2023-01-24T17:00:38.378 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 4.5 MB/s eta 0:00:00 2023-01-24T17:00:38.502 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.46-py3-none-any.whl (132 kB) 2023-01-24T17:00:38.543 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 3.3 MB/s eta 0:00:00 2023-01-24T17:00:38.750 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.45-py3-none-any.whl (132 kB) 2023-01-24T17:00:38.787 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 3.9 MB/s eta 0:00:00 2023-01-24T17:00:38.916 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.44-py3-none-any.whl (132 kB) 2023-01-24T17:00:38.963 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 2.9 MB/s eta 0:00:00 2023-01-24T17:00:39.081 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.43-py3-none-any.whl (132 kB) 2023-01-24T17:00:39.112 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 4.5 MB/s eta 0:00:00 2023-01-24T17:00:39.275 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.42-py3-none-any.whl (132 kB) 2023-01-24T17:00:39.313 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 3.6 MB/s eta 0:00:00 2023-01-24T17:00:39.513 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.41-py3-none-any.whl (132 kB) 2023-01-24T17:00:39.547 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 4.1 MB/s eta 0:00:00 2023-01-24T17:00:39.754 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.40-py3-none-any.whl (132 kB) 2023-01-24T17:00:39.788 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 4.0 MB/s eta 0:00:00 2023-01-24T17:00:39.987 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.39-py3-none-any.whl (132 kB) 2023-01-24T17:00:40.015 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 5.2 MB/s eta 0:00:00 2023-01-24T17:00:40.204 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.38-py3-none-any.whl (132 kB) 2023-01-24T17:00:40.239 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 3.9 MB/s eta 0:00:00 2023-01-24T17:00:40.431 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.37-py3-none-any.whl (132 kB) 2023-01-24T17:00:40.465 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 3.9 MB/s eta 0:00:00 2023-01-24T17:00:40.578 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.36-py3-none-any.whl (132 kB) 2023-01-24T17:00:40.615 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 3.6 MB/s eta 0:00:00 2023-01-24T17:00:40.727 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.35-py3-none-any.whl (132 kB) 2023-01-24T17:00:40.766 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 3.5 MB/s eta 0:00:00 2023-01-24T17:00:40.954 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.34-py3-none-any.whl (132 kB) 2023-01-24T17:00:41.009 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 2.4 MB/s eta 0:00:00 2023-01-24T17:00:41.121 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.33-py3-none-any.whl (132 kB) 2023-01-24T17:00:41.151 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 4.7 MB/s eta 0:00:00 2023-01-24T17:00:41.299 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.32-py3-none-any.whl (132 kB) 2023-01-24T17:00:41.326 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.7/132.7 kB 5.3 MB/s eta 0:00:00 2023-01-24T17:00:41.435 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.31-py3-none-any.whl (132 kB) 2023-01-24T17:00:41.468 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.6/132.6 kB 4.2 MB/s eta 0:00:00 2023-01-24T17:00:41.658 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.30-py3-none-any.whl (132 kB) 2023-01-24T17:00:41.686 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.6/132.6 kB 5.1 MB/s eta 0:00:00 2023-01-24T17:00:41.885 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.29-py3-none-any.whl (132 kB) 2023-01-24T17:00:41.911 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.6/132.6 kB 5.6 MB/s eta 0:00:00 2023-01-24T17:00:42.029 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.28-py3-none-any.whl (132 kB) 2023-01-24T17:00:42.062 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.6/132.6 kB 4.2 MB/s eta 0:00:00 2023-01-24T17:00:42.173 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.27-py3-none-any.whl (132 kB) 2023-01-24T17:00:42.212 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.6/132.6 kB 3.5 MB/s eta 0:00:00 2023-01-24T17:00:42.404 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.26-py3-none-any.whl (132 kB) 2023-01-24T17:00:42.437 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.6/132.6 kB 4.1 MB/s eta 0:00:00 2023-01-24T17:00:42.598 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.25-py3-none-any.whl (132 kB) 2023-01-24T17:00:42.639 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.6/132.6 kB 3.3 MB/s eta 0:00:00 2023-01-24T17:00:42.767 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.24-py3-none-any.whl (132 kB) 2023-01-24T17:00:42.806 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.6/132.6 kB 3.6 MB/s eta 0:00:00 2023-01-24T17:00:42.918 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.23-py3-none-any.whl (132 kB) 2023-01-24T17:00:42.947 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.6/132.6 kB 4.8 MB/s eta 0:00:00 2023-01-24T17:00:43.137 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.22-py3-none-any.whl (132 kB) 2023-01-24T17:00:43.173 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.6/132.6 kB 3.8 MB/s eta 0:00:00 2023-01-24T17:00:43.363 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.21-py3-none-any.whl (132 kB) 2023-01-24T17:00:43.399 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.6/132.6 kB 3.9 MB/s eta 0:00:00 2023-01-24T17:00:43.548 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.20-py3-none-any.whl (132 kB) 2023-01-24T17:00:43.574 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.6/132.6 kB 5.5 MB/s eta 0:00:00 2023-01-24T17:00:43.714 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.19-py3-none-any.whl (132 kB) 2023-01-24T17:00:43.753 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.6/132.6 kB 3.8 MB/s eta 0:00:00 2023-01-24T17:00:43.863 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.18-py3-none-any.whl (132 kB) 2023-01-24T17:00:43.901 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.6/132.6 kB 3.6 MB/s eta 0:00:00 2023-01-24T17:00:44.096 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.17-py3-none-any.whl (132 kB) 2023-01-24T17:00:44.125 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.6/132.6 kB 4.8 MB/s eta 0:00:00 2023-01-24T17:00:44.314 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.16-py3-none-any.whl (132 kB) 2023-01-24T17:00:44.343 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 4.9 MB/s eta 0:00:00 2023-01-24T17:00:44.493 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.15-py3-none-any.whl (132 kB) 2023-01-24T17:00:44.528 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 4.1 MB/s eta 0:00:00 2023-01-24T17:00:44.640 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.14-py3-none-any.whl (132 kB) 2023-01-24T17:00:44.669 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 4.8 MB/s eta 0:00:00 2023-01-24T17:00:44.781 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.13-py3-none-any.whl (132 kB) 2023-01-24T17:00:44.820 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 3.5 MB/s eta 0:00:00 2023-01-24T17:00:45.016 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.12-py3-none-any.whl (132 kB) 2023-01-24T17:00:45.046 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 4.9 MB/s eta 0:00:00 2023-01-24T17:00:45.270 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.11-py3-none-any.whl (132 kB) 2023-01-24T17:00:45.312 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 3.2 MB/s eta 0:00:00 2023-01-24T17:00:45.423 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.10-py3-none-any.whl (132 kB) 2023-01-24T17:00:45.455 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 4.5 MB/s eta 0:00:00 2023-01-24T17:00:45.606 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.9-py3-none-any.whl (132 kB) 2023-01-24T17:00:45.647 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 3.3 MB/s eta 0:00:00 2023-01-24T17:00:45.808 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.8-py3-none-any.whl (132 kB) 2023-01-24T17:00:45.857 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 2.7 MB/s eta 0:00:00 2023-01-24T17:00:46.008 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.7-py3-none-any.whl (132 kB) 2023-01-24T17:00:46.040 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 4.3 MB/s eta 0:00:00 2023-01-24T17:00:46.157 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.6-py3-none-any.whl (132 kB) 2023-01-24T17:00:46.185 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 5.0 MB/s eta 0:00:00 2023-01-24T17:00:46.375 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.5-py3-none-any.whl (132 kB) 2023-01-24T17:00:46.410 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 3.9 MB/s eta 0:00:00 2023-01-24T17:00:46.600 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.4-py3-none-any.whl (132 kB) 2023-01-24T17:00:46.635 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 3.9 MB/s eta 0:00:00 2023-01-24T17:00:46.749 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.3-py3-none-any.whl (132 kB) 2023-01-24T17:00:46.786 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 3.6 MB/s eta 0:00:00 2023-01-24T17:00:46.978 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.2-py3-none-any.whl (132 kB) 2023-01-24T17:00:47.011 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 4.1 MB/s eta 0:00:00 2023-01-24T17:00:47.198 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.1-py3-none-any.whl (132 kB) 2023-01-24T17:00:47.230 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 4.5 MB/s eta 0:00:00 2023-01-24T17:00:47.385 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.26.0-py3-none-any.whl (132 kB) 2023-01-24T17:00:47.422 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 3.8 MB/s eta 0:00:00 2023-01-24T17:00:47.648 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.25.5-py3-none-any.whl (132 kB) 2023-01-24T17:00:47.689 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 3.4 MB/s eta 0:00:00 2023-01-24T17:00:47.829 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.25.4-py3-none-any.whl (132 kB) 2023-01-24T17:00:47.864 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 4.1 MB/s eta 0:00:00 2023-01-24T17:00:47.988 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.25.3-py3-none-any.whl (132 kB) 2023-01-24T17:00:48.023 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 4.0 MB/s eta 0:00:00 2023-01-24T17:00:48.226 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.25.2-py3-none-any.whl (132 kB) 2023-01-24T17:00:48.265 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 3.5 MB/s eta 0:00:00 2023-01-24T17:00:48.442 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.25.1-py3-none-any.whl (132 kB) 2023-01-24T17:00:48.473 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 4.4 MB/s eta 0:00:00 2023-01-24T17:00:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.25.0-py3-none-any.whl (132 kB) 2023-01-24T17:00:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 4.4 MB/s eta 0:00:00 2023-01-24T17:00:48.840 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading boto3-1.24.96-py3-none-any.whl (132 kB) 2023-01-24T17:00:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 132.5/132.5 kB 4.0 MB/s eta 0:00:00 2023-01-24T17:00:49.096 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting python-dateutil<3.0.0,>=2.1 2023-01-24T17:00:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-01-24T17:00:49.183 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 247.7/247.7 kB 6.3 MB/s eta 0:00:00 2023-01-24T17:00:49.309 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting urllib3<1.27,>=1.25.4 2023-01-24T17:00:49.310 INFO:teuthology.orchestra.run.smithi103.stdout: Using cached urllib3-1.26.14-py2.py3-none-any.whl (140 kB) 2023-01-24T17:00:49.386 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting six 2023-01-24T17:00:49.388 INFO:teuthology.orchestra.run.smithi103.stdout: Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-01-24T17:00:49.498 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting zope.event 2023-01-24T17:00:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading zope.event-4.6-py2.py3-none-any.whl (6.8 kB) 2023-01-24T17:00:49.798 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting greenlet>=2.0.0 2023-01-24T17:00:49.800 INFO:teuthology.orchestra.run.smithi103.stdout: Using cached greenlet-2.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (544 kB) 2023-01-24T17:00:49.803 INFO:teuthology.orchestra.run.smithi103.stdout:Requirement already satisfied: setuptools in ./virtualenv/lib/python3.8/site-packages (from gevent>=1.0->-r requirements.txt (line 9)) (44.1.0) 2023-01-24T17:00:50.030 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting zope.interface 2023-01-24T17:00:50.088 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading zope.interface-5.5.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (261 kB) 2023-01-24T17:00:50.129 INFO:teuthology.orchestra.run.smithi103.stdout: --------------------------------------- 261.4/261.4 kB 6.8 MB/s eta 0:00:00 2023-01-24T17:00:50.235 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting certifi>=2017.4.17 2023-01-24T17:00:50.237 INFO:teuthology.orchestra.run.smithi103.stdout: Using cached certifi-2022.12.7-py3-none-any.whl (155 kB) 2023-01-24T17:00:50.305 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting idna<4,>=2.5 2023-01-24T17:00:50.306 INFO:teuthology.orchestra.run.smithi103.stdout: Using cached idna-3.4-py3-none-any.whl (61 kB) 2023-01-24T17:00:50.467 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting charset-normalizer<4,>=2 2023-01-24T17:00:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: Using cached charset_normalizer-3.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (195 kB) 2023-01-24T17:00:50.584 INFO:teuthology.orchestra.run.smithi103.stdout:Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 2023-01-24T17:00:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB) 2023-01-24T17:00:50.649 INFO:teuthology.orchestra.run.smithi103.stdout: ---------------------------------------- 98.3/98.3 kB 2.3 MB/s eta 0:00:00 2023-01-24T17:00:51.017 INFO:teuthology.orchestra.run.smithi103.stdout:Installing collected packages: pytz, nose, charset-normalizer, boto, zope.interface, zope.event, urllib3, six, PyYAML, pyparsing, lxml, jmespath, idna, greenlet, certifi, requests, python-dateutil, munch, isodate, httplib2, gevent, botocore, s3transfer, boto3 2023-01-24T17:00:54.693 INFO:teuthology.orchestra.run.smithi103.stdout:Successfully installed PyYAML-6.0 boto-2.49.0 boto3-1.24.96 botocore-1.27.96 certifi-2022.12.7 charset-normalizer-3.0.1 gevent-22.10.2 greenlet-2.0.1 httplib2-0.21.0 idna-3.4 isodate-0.6.1 jmespath-1.0.1 lxml-4.9.2 munch-2.5.0 nose-1.3.7 pyparsing-3.0.9 python-dateutil-2.8.2 pytz-2022.7.1 requests-2.28.2 s3transfer-0.6.0 six-1.16.0 urllib3-1.26.14 zope.event-4.6 zope.interface-5.5.2 2023-01-24T17:00:55.052 INFO:teuthology.orchestra.run.smithi103.stdout:running develop 2023-01-24T17:00:55.073 INFO:teuthology.orchestra.run.smithi103.stdout:running egg_info 2023-01-24T17:00:55.074 INFO:teuthology.orchestra.run.smithi103.stdout:creating s3tests.egg-info 2023-01-24T17:00:55.074 INFO:teuthology.orchestra.run.smithi103.stdout:writing s3tests.egg-info/PKG-INFO 2023-01-24T17:00:55.074 INFO:teuthology.orchestra.run.smithi103.stdout:writing dependency_links to s3tests.egg-info/dependency_links.txt 2023-01-24T17:00:55.075 INFO:teuthology.orchestra.run.smithi103.stdout:writing requirements to s3tests.egg-info/requires.txt 2023-01-24T17:00:55.075 INFO:teuthology.orchestra.run.smithi103.stdout:writing top-level names to s3tests.egg-info/top_level.txt 2023-01-24T17:00:55.075 INFO:teuthology.orchestra.run.smithi103.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-01-24T17:00:55.104 INFO:teuthology.orchestra.run.smithi103.stdout:reading manifest file 's3tests.egg-info/SOURCES.txt' 2023-01-24T17:00:55.105 INFO:teuthology.orchestra.run.smithi103.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2023-01-24T17:00:55.105 INFO:teuthology.orchestra.run.smithi103.stdout:running build_ext 2023-01-24T17:00:55.105 INFO:teuthology.orchestra.run.smithi103.stdout:Creating /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages/s3tests.egg-link (link to .) 2023-01-24T17:00:55.106 INFO:teuthology.orchestra.run.smithi103.stdout:Adding s3tests 0.0.1 to easy-install.pth file 2023-01-24T17:00:55.106 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.107 INFO:teuthology.orchestra.run.smithi103.stdout:Installed /home/ubuntu/cephtest/s3-tests 2023-01-24T17:00:55.107 INFO:teuthology.orchestra.run.smithi103.stdout:Processing dependencies for s3tests==0.0.1 2023-01-24T17:00:55.184 INFO:teuthology.orchestra.run.smithi103.stdout:Searching for isodate==0.6.1 2023-01-24T17:00:55.185 INFO:teuthology.orchestra.run.smithi103.stdout:Best match: isodate 0.6.1 2023-01-24T17:00:55.185 INFO:teuthology.orchestra.run.smithi103.stdout:Adding isodate 0.6.1 to easy-install.pth file 2023-01-24T17:00:55.185 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.185 INFO:teuthology.orchestra.run.smithi103.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-01-24T17:00:55.187 INFO:teuthology.orchestra.run.smithi103.stdout:Searching for gevent==22.10.2 2023-01-24T17:00:55.187 INFO:teuthology.orchestra.run.smithi103.stdout:Best match: gevent 22.10.2 2023-01-24T17:00:55.187 INFO:teuthology.orchestra.run.smithi103.stdout:Adding gevent 22.10.2 to easy-install.pth file 2023-01-24T17:00:55.187 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.187 INFO:teuthology.orchestra.run.smithi103.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-01-24T17:00:55.188 INFO:teuthology.orchestra.run.smithi103.stdout:Searching for munch==2.5.0 2023-01-24T17:00:55.188 INFO:teuthology.orchestra.run.smithi103.stdout:Best match: munch 2.5.0 2023-01-24T17:00:55.188 INFO:teuthology.orchestra.run.smithi103.stdout:Adding munch 2.5.0 to easy-install.pth file 2023-01-24T17:00:55.188 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.188 INFO:teuthology.orchestra.run.smithi103.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-01-24T17:00:55.189 INFO:teuthology.orchestra.run.smithi103.stdout:Searching for PyYAML==6.0 2023-01-24T17:00:55.189 INFO:teuthology.orchestra.run.smithi103.stdout:Best match: PyYAML 6.0 2023-01-24T17:00:55.191 INFO:teuthology.orchestra.run.smithi103.stdout:Adding PyYAML 6.0 to easy-install.pth file 2023-01-24T17:00:55.191 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.191 INFO:teuthology.orchestra.run.smithi103.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-01-24T17:00:55.192 INFO:teuthology.orchestra.run.smithi103.stdout:Searching for boto3==1.24.96 2023-01-24T17:00:55.192 INFO:teuthology.orchestra.run.smithi103.stdout:Best match: boto3 1.24.96 2023-01-24T17:00:55.192 INFO:teuthology.orchestra.run.smithi103.stdout:Adding boto3 1.24.96 to easy-install.pth file 2023-01-24T17:00:55.192 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.192 INFO:teuthology.orchestra.run.smithi103.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-01-24T17:00:55.193 INFO:teuthology.orchestra.run.smithi103.stdout:Searching for boto==2.49.0 2023-01-24T17:00:55.193 INFO:teuthology.orchestra.run.smithi103.stdout:Best match: boto 2.49.0 2023-01-24T17:00:55.193 INFO:teuthology.orchestra.run.smithi103.stdout:Adding boto 2.49.0 to easy-install.pth file 2023-01-24T17:00:55.193 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.193 INFO:teuthology.orchestra.run.smithi103.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-01-24T17:00:55.194 INFO:teuthology.orchestra.run.smithi103.stdout:Searching for six==1.16.0 2023-01-24T17:00:55.194 INFO:teuthology.orchestra.run.smithi103.stdout:Best match: six 1.16.0 2023-01-24T17:00:55.194 INFO:teuthology.orchestra.run.smithi103.stdout:Adding six 1.16.0 to easy-install.pth file 2023-01-24T17:00:55.194 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.194 INFO:teuthology.orchestra.run.smithi103.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-01-24T17:00:55.194 INFO:teuthology.orchestra.run.smithi103.stdout:Searching for setuptools==44.1.0 2023-01-24T17:00:55.195 INFO:teuthology.orchestra.run.smithi103.stdout:Best match: setuptools 44.1.0 2023-01-24T17:00:55.195 INFO:teuthology.orchestra.run.smithi103.stdout:Adding setuptools 44.1.0 to easy-install.pth file 2023-01-24T17:00:55.195 INFO:teuthology.orchestra.run.smithi103.stdout:Installing easy_install script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2023-01-24T17:00:55.195 INFO:teuthology.orchestra.run.smithi103.stdout:Installing easy_install-3.8 script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2023-01-24T17:00:55.195 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.196 INFO:teuthology.orchestra.run.smithi103.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-01-24T17:00:55.197 INFO:teuthology.orchestra.run.smithi103.stdout:Searching for zope.event==4.6 2023-01-24T17:00:55.197 INFO:teuthology.orchestra.run.smithi103.stdout:Best match: zope.event 4.6 2023-01-24T17:00:55.198 INFO:teuthology.orchestra.run.smithi103.stdout:Adding zope.event 4.6 to easy-install.pth file 2023-01-24T17:00:55.198 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.198 INFO:teuthology.orchestra.run.smithi103.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-01-24T17:00:55.199 INFO:teuthology.orchestra.run.smithi103.stdout:Searching for zope.interface==5.5.2 2023-01-24T17:00:55.200 INFO:teuthology.orchestra.run.smithi103.stdout:Best match: zope.interface 5.5.2 2023-01-24T17:00:55.200 INFO:teuthology.orchestra.run.smithi103.stdout:Adding zope.interface 5.5.2 to easy-install.pth file 2023-01-24T17:00:55.200 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.200 INFO:teuthology.orchestra.run.smithi103.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-01-24T17:00:55.200 INFO:teuthology.orchestra.run.smithi103.stdout:Searching for greenlet==2.0.1 2023-01-24T17:00:55.201 INFO:teuthology.orchestra.run.smithi103.stdout:Best match: greenlet 2.0.1 2023-01-24T17:00:55.201 INFO:teuthology.orchestra.run.smithi103.stdout:Adding greenlet 2.0.1 to easy-install.pth file 2023-01-24T17:00:55.201 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.201 INFO:teuthology.orchestra.run.smithi103.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-01-24T17:00:55.201 INFO:teuthology.orchestra.run.smithi103.stdout:Searching for s3transfer==0.6.0 2023-01-24T17:00:55.202 INFO:teuthology.orchestra.run.smithi103.stdout:Best match: s3transfer 0.6.0 2023-01-24T17:00:55.202 INFO:teuthology.orchestra.run.smithi103.stdout:Adding s3transfer 0.6.0 to easy-install.pth file 2023-01-24T17:00:55.202 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.202 INFO:teuthology.orchestra.run.smithi103.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-01-24T17:00:55.204 INFO:teuthology.orchestra.run.smithi103.stdout:Searching for jmespath==1.0.1 2023-01-24T17:00:55.204 INFO:teuthology.orchestra.run.smithi103.stdout:Best match: jmespath 1.0.1 2023-01-24T17:00:55.204 INFO:teuthology.orchestra.run.smithi103.stdout:Adding jmespath 1.0.1 to easy-install.pth file 2023-01-24T17:00:55.205 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.205 INFO:teuthology.orchestra.run.smithi103.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-01-24T17:00:55.205 INFO:teuthology.orchestra.run.smithi103.stdout:Searching for botocore==1.27.96 2023-01-24T17:00:55.206 INFO:teuthology.orchestra.run.smithi103.stdout:Best match: botocore 1.27.96 2023-01-24T17:00:55.206 INFO:teuthology.orchestra.run.smithi103.stdout:Adding botocore 1.27.96 to easy-install.pth file 2023-01-24T17:00:55.206 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.206 INFO:teuthology.orchestra.run.smithi103.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-01-24T17:00:55.207 INFO:teuthology.orchestra.run.smithi103.stdout:Searching for python-dateutil==2.8.2 2023-01-24T17:00:55.207 INFO:teuthology.orchestra.run.smithi103.stdout:Best match: python-dateutil 2.8.2 2023-01-24T17:00:55.207 INFO:teuthology.orchestra.run.smithi103.stdout:Adding python-dateutil 2.8.2 to easy-install.pth file 2023-01-24T17:00:55.207 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.207 INFO:teuthology.orchestra.run.smithi103.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-01-24T17:00:55.207 INFO:teuthology.orchestra.run.smithi103.stdout:Searching for urllib3==1.26.14 2023-01-24T17:00:55.208 INFO:teuthology.orchestra.run.smithi103.stdout:Best match: urllib3 1.26.14 2023-01-24T17:00:55.208 INFO:teuthology.orchestra.run.smithi103.stdout:Adding urllib3 1.26.14 to easy-install.pth file 2023-01-24T17:00:55.208 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:00:55.208 INFO:teuthology.orchestra.run.smithi103.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python3.8/site-packages 2023-01-24T17:00:55.208 INFO:teuthology.orchestra.run.smithi103.stdout:Finished processing dependencies for s3tests==0.0.1 2023-01-24T17:00:55.219 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T17:00:55.219 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2023-01-24T17:00:55.226 INFO:tasks.s3tests:Configuring boto... 2023-01-24T17:00:55.226 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2023-01-24T17:00:55.227 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/boto.cfg 2023-01-24T17:00:55.273 DEBUG:teuthology.orchestra.run.smithi103:s3 tests against rgw> S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain,!sse-s3' 2023-01-24T17:00:55.722 INFO:teuthology.orchestra.run.smithi103.stderr:/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_headers.py:126: SyntaxWarning: "is" with a literal. Did you mean "=="? 2023-01-24T17:00:55.723 INFO:teuthology.orchestra.run.smithi103.stderr: if boto_type is 'S3Connection': 2023-01-24T17:00:55.723 INFO:teuthology.orchestra.run.smithi103.stderr:/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_headers.py:131: SyntaxWarning: "is" with a literal. Did you mean "=="? 2023-01-24T17:00:55.723 INFO:teuthology.orchestra.run.smithi103.stderr: elif boto_type is 'HTTPRequest': 2023-01-24T17:00:55.723 INFO:teuthology.orchestra.run.smithi103.stderr:/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3_website.py:192: SyntaxWarning: "is" with a literal. Did you mean "=="? 2023-01-24T17:00:55.723 INFO:teuthology.orchestra.run.smithi103.stderr: if k is 'BucketName': 2023-01-24T17:00:55.742 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok 2023-01-24T17:00:55.758 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok 2023-01-24T17:00:55.782 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_date_and_amz_date ... ok 2023-01-24T17:00:55.848 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok 2023-01-24T17:00:55.873 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_none ... ok 2023-01-24T17:00:55.898 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_contentlength_none ... ok 2023-01-24T17:00:55.933 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_acl_create_contentlength_none ... ok 2023-01-24T17:00:55.951 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok 2023-01-24T17:00:55.996 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_empty ... ok 2023-01-24T17:00:55.999 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_none ... ok 2023-01-24T17:00:56.023 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws2 ... ok 2023-01-24T17:00:56.042 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws2 ... ok 2023-01-24T17:00:56.063 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws2 ... ok 2023-01-24T17:00:56.083 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws2 ... ok 2023-01-24T17:00:56.085 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws2 ... ok 2023-01-24T17:00:56.087 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws2 ... ok 2023-01-24T17:00:56.088 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws4 ... SKIP 2023-01-24T17:00:56.089 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws4 ... SKIP 2023-01-24T17:00:56.089 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws4 ... SKIP 2023-01-24T17:00:56.089 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws4 ... SKIP 2023-01-24T17:00:56.089 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_empty_aws4 ... SKIP 2023-01-24T17:00:56.093 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_none_aws4 ... SKIP 2023-01-24T17:00:56.093 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_date_invalid_aws4 ... SKIP 2023-01-24T17:00:56.093 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_invalid_aws4 ... SKIP 2023-01-24T17:00:56.093 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_date_empty_aws4 ... SKIP 2023-01-24T17:00:56.093 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_empty_aws4 ... SKIP 2023-01-24T17:00:56.094 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws4 ... SKIP 2023-01-24T17:00:56.094 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_none_aws4 ... SKIP 2023-01-24T17:00:56.094 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_today_aws4 ... SKIP 2023-01-24T17:00:56.094 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_today_aws4 ... SKIP 2023-01-24T17:00:56.094 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_today_aws4 ... SKIP 2023-01-24T17:00:56.095 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_today_aws4 ... SKIP 2023-01-24T17:00:56.095 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_epoch_aws4 ... SKIP 2023-01-24T17:00:56.095 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-01-24T17:00:56.095 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_end_aws4 ... SKIP 2023-01-24T17:00:56.095 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_end_aws4 ... SKIP 2023-01-24T17:00:56.096 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_custom_header_aws4 ... SKIP 2023-01-24T17:00:56.096 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_header_aws4 ... SKIP 2023-01-24T17:00:56.096 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws4 ... SKIP 2023-01-24T17:00:56.096 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_empty_aws4 ... SKIP 2023-01-24T17:00:56.096 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_none_aws4 ... SKIP 2023-01-24T17:00:56.097 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_invalid_aws4 ... SKIP 2023-01-24T17:00:56.097 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_invalid_aws4 ... SKIP 2023-01-24T17:00:56.097 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_empty_aws4 ... SKIP 2023-01-24T17:00:56.097 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_empty_aws4 ... SKIP 2023-01-24T17:00:56.098 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws4 ... SKIP 2023-01-24T17:00:56.098 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_none_aws4 ... SKIP 2023-01-24T17:00:56.098 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_today_aws4 ... SKIP 2023-01-24T17:00:56.099 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_today_aws4 ... SKIP 2023-01-24T17:00:56.099 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_after_today_aws4 ... SKIP 2023-01-24T17:00:56.099 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_after_today_aws4 ... SKIP 2023-01-24T17:00:56.099 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws4 ... SKIP 2023-01-24T17:00:56.100 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_epoch_aws4 ... SKIP 2023-01-24T17:00:56.140 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3.test_versioning_obj_read_not_exist_null ... ok 2023-01-24T17:00:56.141 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3.test_object_storage_class ... SKIP 2023-01-24T17:00:56.141 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3.test_object_storage_class_multipart ... SKIP 2023-01-24T17:00:56.141 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3.test_object_modify_storage_class ... SKIP 2023-01-24T17:00:56.142 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3.test_object_modify_storage_class_multipart ... SKIP 2023-01-24T17:00:56.142 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3.test_object_storage_class_copy ... SKIP 2023-01-24T17:00:56.143 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3.test_object_storage_class_copy_multipart ... SKIP 2023-01-24T17:00:56.244 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3.test_atomic_dual_conditional_write_1mb ... ok 2023-01-24T17:00:56.336 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3.test_atomic_write_bucket_gone ... ok 2023-01-24T17:00:57.802 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_1 ... ok 2023-01-24T17:00:57.866 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_2 ... ok 2023-01-24T17:00:57.914 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3.test_bucket_policy_set_condition_operator_end_with_IfExists ... ok 2023-01-24T17:00:57.946 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_request_obj_tag ... ok 2023-01-24T17:00:58.085 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.check_can_test_website ... SKIP 2023-01-24T17:00:58.085 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_nonexistant_bucket_rgw ... SKIP 2023-01-24T17:00:58.086 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_public_index ... SKIP 2023-01-24T17:00:58.086 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_public_index ... SKIP 2023-01-24T17:00:58.086 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty ... SKIP 2023-01-24T17:00:58.086 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty ... SKIP 2023-01-24T17:00:58.087 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index ... SKIP 2023-01-24T17:00:58.087 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index ... SKIP 2023-01-24T17:00:58.087 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_missingerrordoc ... SKIP 2023-01-24T17:00:58.087 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_missingerrordoc ... SKIP 2023-01-24T17:00:58.087 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_missingerrordoc ... SKIP 2023-01-24T17:00:58.088 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_missingerrordoc ... SKIP 2023-01-24T17:00:58.088 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_blockederrordoc ... SKIP 2023-01-24T17:00:58.088 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_pubilc_errordoc ... SKIP 2023-01-24T17:00:58.088 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_blockederrordoc ... SKIP 2023-01-24T17:00:58.089 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_blockederrordoc ... SKIP 2023-01-24T17:00:58.089 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_blockederrordoc ... SKIP 2023-01-24T17:00:58.089 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_gooderrordoc ... SKIP 2023-01-24T17:00:58.089 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_gooderrordoc ... SKIP 2023-01-24T17:00:58.089 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_gooderrordoc ... SKIP 2023-01-24T17:00:58.091 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_gooderrordoc ... SKIP 2023-01-24T17:00:58.092 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_base ... SKIP 2023-01-24T17:00:58.092 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path ... SKIP 2023-01-24T17:00:58.093 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path_upgrade ... SKIP 2023-01-24T17:00:58.093 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_xredirect_nonwebsite ... SKIP 2023-01-24T17:00:58.093 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_relative ... SKIP 2023-01-24T17:00:58.093 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_abs ... SKIP 2023-01-24T17:00:58.093 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_relative ... SKIP 2023-01-24T17:00:58.094 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_abs ... SKIP 2023-01-24T17:00:58.094 INFO:teuthology.orchestra.run.smithi103.stderr:SKIP 2023-01-24T17:01:00.306 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests.functional.test_utils.test_generate ... ok 2023-01-24T17:01:00.343 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok 2023-01-24T17:01:07.054 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_bad ... ok 2023-01-24T17:01:07.094 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_empty ... ok 2023-01-24T17:01:07.156 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_none ... ok 2023-01-24T17:01:07.221 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_mismatch ... ok 2023-01-24T17:01:07.288 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_empty ... ok 2023-01-24T17:01:07.346 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_none ... ok 2023-01-24T17:01:07.381 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contentlength_negative ... ok 2023-01-24T17:01:07.439 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok 2023-01-24T17:01:07.480 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_empty ... ok 2023-01-24T17:01:07.554 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_none ... ok 2023-01-24T17:01:07.594 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_bucket_put_bad_canned_acl ... ok 2023-01-24T17:01:07.621 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_mismatch ... ok 2023-01-24T17:01:07.648 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_empty ... ok 2023-01-24T17:01:07.659 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_contentlength_negative ... ok 2023-01-24T17:01:07.693 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws2 ... ok 2023-01-24T17:01:07.742 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_empty_aws2 ... ok 2023-01-24T17:01:07.795 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_none_aws2 ... ok 2023-01-24T17:01:07.836 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_invalid_aws2 ... ok 2023-01-24T17:01:07.881 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_empty_aws2 ... ok 2023-01-24T17:01:07.920 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_today_aws2 ... ok 2023-01-24T17:01:07.957 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_epoch_aws2 ... ok 2023-01-24T17:01:07.994 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_after_end_aws2 ... ok 2023-01-24T17:01:08.017 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_empty_aws2 ... ok 2023-01-24T17:01:08.041 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_none_aws2 ... ok 2023-01-24T17:01:08.050 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_invalid_aws2 ... ok 2023-01-24T17:01:08.061 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_empty_aws2 ... ok 2023-01-24T17:01:08.069 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_today_aws2 ... ok 2023-01-24T17:01:08.078 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_after_today_aws2 ... ok 2023-01-24T17:01:08.086 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws2 ... ok 2023-01-24T17:01:08.136 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_empty ... ok 2023-01-24T17:01:08.205 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_distinct ... ok 2023-01-24T17:01:08.290 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_many ... ok 2023-01-24T17:01:08.381 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_many ... ok 2023-01-24T17:01:08.506 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_basic_key_count ... ok 2023-01-24T17:01:08.599 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_basic ... ok 2023-01-24T17:01:08.693 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_basic ... ok 2023-01-24T17:01:08.783 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_encoding_basic ... ok 2023-01-24T17:01:08.907 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_encoding_basic ... ok 2023-01-24T17:01:09.076 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix ... ok 2023-01-24T17:01:09.253 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix ... ok 2023-01-24T17:01:09.304 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_ends_with_delimiter ... ok 2023-01-24T17:01:09.355 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_ends_with_delimiter ... ok 2023-01-24T17:01:09.450 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_alt ... ok 2023-01-24T17:01:09.538 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_alt ... ok 2023-01-24T17:01:09.723 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_underscore ... ok 2023-01-24T17:01:09.896 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_underscore ... ok 2023-01-24T17:01:10.015 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_percentage ... ok 2023-01-24T17:01:10.100 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_percentage ... ok 2023-01-24T17:01:10.185 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_whitespace ... ok 2023-01-24T17:01:10.278 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_whitespace ... ok 2023-01-24T17:01:10.368 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_dot ... ok 2023-01-24T17:01:10.459 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_dot ... ok 2023-01-24T17:01:10.549 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_unreadable ... ok 2023-01-24T17:01:10.648 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_unreadable ... ok 2023-01-24T17:01:10.738 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_empty ... ok 2023-01-24T17:01:10.830 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_empty ... ok 2023-01-24T17:01:10.921 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_none ... ok 2023-01-24T17:01:11.016 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_none ... ok 2023-01-24T17:01:11.098 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_notempty ... ok 2023-01-24T17:01:11.177 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_defaultempty ... ok 2023-01-24T17:01:11.251 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_empty ... ok 2023-01-24T17:01:11.363 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_exist ... ok 2023-01-24T17:01:11.451 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_not_exist ... ok 2023-01-24T17:01:23.791 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_skip_special ... ok 2023-01-24T17:01:23.868 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_basic ... ok 2023-01-24T17:01:23.947 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_basic ... ok 2023-01-24T17:01:24.054 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_alt ... ok 2023-01-24T17:01:24.134 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_alt ... ok 2023-01-24T17:01:24.212 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_empty ... ok 2023-01-24T17:01:24.299 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_empty ... ok 2023-01-24T17:01:24.383 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_none ... ok 2023-01-24T17:01:24.464 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_none ... ok 2023-01-24T17:01:24.543 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_not_exist ... ok 2023-01-24T17:01:24.625 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_not_exist ... ok 2023-01-24T17:01:24.703 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_unreadable ... ok 2023-01-24T17:01:24.784 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_unreadable ... ok 2023-01-24T17:01:24.879 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_basic ... ok 2023-01-24T17:01:24.971 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_basic ... ok 2023-01-24T17:01:25.067 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_alt ... ok 2023-01-24T17:01:25.169 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_alt ... ok 2023-01-24T17:01:25.260 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_not_exist ... ok 2023-01-24T17:01:25.383 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_not_exist ... ok 2023-01-24T17:01:25.459 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_delimiter_not_exist ... ok 2023-01-24T17:01:25.552 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_delimiter_not_exist ... ok 2023-01-24T17:01:25.642 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-01-24T17:01:25.727 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist ... ok 2023-01-24T17:01:25.826 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_one ... ok 2023-01-24T17:01:25.921 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_one ... ok 2023-01-24T17:01:26.014 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_zero ... ok 2023-01-24T17:01:26.107 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_zero ... ok 2023-01-24T17:01:26.199 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_none ... ok 2023-01-24T17:01:26.294 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_none ... ok 2023-01-24T17:01:26.687 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_account_usage ... ok 2023-01-24T17:01:26.735 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_head_bucket_usage ... ok 2023-01-24T17:01:27.173 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_unordered ... ok 2023-01-24T17:01:27.573 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_unordered ... ok 2023-01-24T17:01:27.660 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_invalid ... ok 2023-01-24T17:01:27.751 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_none ... ok 2023-01-24T17:01:27.848 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_empty ... ok 2023-01-24T17:01:27.932 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken_empty ... ok 2023-01-24T17:01:28.034 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken ... ok 2023-01-24T17:01:28.138 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_both_continuationtoken_startafter ... ok 2023-01-24T17:01:28.239 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_unreadable ... ok 2023-01-24T17:01:28.335 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_unreadable ... ok 2023-01-24T17:01:28.431 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_not_in_list ... ok 2023-01-24T17:01:28.531 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_not_in_list ... ok 2023-01-24T17:01:28.616 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_after_list ... ok 2023-01-24T17:01:28.702 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_after_list ... ok 2023-01-24T17:01:28.819 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data ... ok 2023-01-24T17:01:29.053 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data_versioning ... ok 2023-01-24T17:01:29.141 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous ... ok 2023-01-24T17:01:29.194 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous ... ok 2023-01-24T17:01:29.232 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous_fail ... ok 2023-01-24T17:01:29.269 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous_fail ... ok 2023-01-24T17:01:29.280 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_notexist ... ok 2023-01-24T17:01:29.293 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_notexist ... ok 2023-01-24T17:01:29.303 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_notexist ... ok 2023-01-24T17:01:29.364 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_nonempty ... ok 2023-01-24T17:01:29.575 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_concurrent_set_canned_acl ... ok 2023-01-24T17:01:29.587 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_write_to_nonexist_bucket ... ok 2023-01-24T17:01:29.694 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_delete ... ok 2023-01-24T17:01:29.733 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_read_not_exist ... ok 2023-01-24T17:01:29.770 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_requestid_matches_header_on_error ... ok 2023-01-24T17:01:29.891 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete ... ok 2023-01-24T17:01:30.065 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete ... ok 2023-01-24T17:01:42.741 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete_key_limit ... ok 2023-01-24T17:01:55.391 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete_key_limit ... ok 2023-01-24T17:01:55.439 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_head_zero_bytes ... ok 2023-01-24T17:01:55.480 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_write_check_etag ... ok 2023-01-24T17:01:55.533 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_write_cache_control ... ok 2023-01-24T17:01:55.589 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_write_expires ... ok 2023-01-24T17:01:55.674 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_write_read_update_read_delete ... ok 2023-01-24T17:01:55.731 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_good ... ok 2023-01-24T17:01:55.781 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_empty ... ok 2023-01-24T17:01:55.859 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_overwrite_to_empty ... ok 2023-01-24T17:01:55.924 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_metadata_replaced_on_put ... ok 2023-01-24T17:01:55.983 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_write_file ... ok 2023-01-24T17:01:56.033 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_anonymous_request ... ok 2023-01-24T17:01:56.085 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request ... ok 2023-01-24T17:01:56.127 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_no_content_type ... ok 2023-01-24T17:01:56.159 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request_bad_access_key ... ok 2023-01-24T17:01:56.193 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_success_code ... ok 2023-01-24T17:01:56.228 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_invalid_success_code ... ok 2023-01-24T17:01:56.333 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_larger_than_chunk ... ok 2023-01-24T17:01:56.383 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_key_from_filename ... ok 2023-01-24T17:01:56.426 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_ignored_header ... ok 2023-01-24T17:01:56.475 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_case_insensitive_condition_fields ... ok 2023-01-24T17:01:56.532 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_escaped_field_values ... ok 2023-01-24T17:01:56.587 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_success_redirect_action ... ok 2023-01-24T17:01:56.623 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_signature ... ok 2023-01-24T17:01:56.657 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_access_key ... ok 2023-01-24T17:01:56.689 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_date_format ... ok 2023-01-24T17:01:56.722 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_no_key_specified ... ok 2023-01-24T17:01:56.754 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_signature ... ok 2023-01-24T17:01:56.789 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_policy_condition ... ok 2023-01-24T17:01:56.841 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_user_specified_header ... ok 2023-01-24T17:01:56.881 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_request_missing_policy_specified_field ... ok 2023-01-24T17:01:56.918 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_condition_is_case_sensitive ... ok 2023-01-24T17:01:56.960 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_expires_is_case_sensitive ... ok 2023-01-24T17:01:56.994 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_expired_policy ... ok 2023-01-24T17:01:57.029 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_request_field_value ... ok 2023-01-24T17:01:57.066 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_expires_condition ... ok 2023-01-24T17:01:57.100 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_conditions_list ... ok 2023-01-24T17:01:57.136 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_limit_exceeded ... ok 2023-01-24T17:01:57.175 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_content_length_argument ... ok 2023-01-24T17:01:57.216 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_content_length_argument ... ok 2023-01-24T17:01:57.253 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_below_minimum ... ok 2023-01-24T17:01:57.296 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_empty_conditions ... ok 2023-01-24T17:01:57.346 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_good ... ok 2023-01-24T17:01:57.396 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_failed ... ok 2023-01-24T17:01:57.453 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_good ... ok 2023-01-24T17:01:57.504 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_failed ... ok 2023-01-24T17:01:57.556 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_good ... ok 2023-01-24T17:01:58.626 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_failed ... ok 2023-01-24T17:01:58.684 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_good ... ok 2023-01-24T17:01:58.770 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_failed ... ok 2023-01-24T17:01:58.879 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_good ... ok 2023-01-24T17:01:58.950 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_failed ... ok 2023-01-24T17:01:59.017 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_overwrite_existed_good ... ok 2023-01-24T17:01:59.066 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_nonexisted_failed ... ok 2023-01-24T17:01:59.138 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_good ... ok 2023-01-24T17:01:59.199 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_failed ... ok 2023-01-24T17:01:59.255 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_nonexisted_good ... ok 2023-01-24T17:01:59.362 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_overwrite_existed_failed ... ok 2023-01-24T17:01:59.412 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get ... ok 2023-01-24T17:01:59.551 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_gone ... ok 2023-01-24T17:01:59.688 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_delete_key_bucket_gone ... ok 2023-01-24T17:01:59.757 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_gone ... ok 2023-01-24T17:01:59.793 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_head ... ok 2023-01-24T17:01:59.802 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_notexist ... ok 2023-01-24T17:01:59.894 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_extended ... ok 2023-01-24T17:01:59.943 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_acl ... ok 2023-01-24T17:02:00.001 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_acl ... ok 2023-01-24T17:02:00.056 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated ... ok 2023-01-24T17:02:00.109 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_raw_response_headers ... ok 2023-01-24T17:02:00.161 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_acl ... ok 2023-01-24T17:02:00.213 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_acl ... ok 2023-01-24T17:02:00.356 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_gone ... ok 2023-01-24T17:02:00.424 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_gone ... ok 2023-01-24T17:02:00.475 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired ... ok 2023-01-24T17:02:00.522 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_range_zero ... ok 2023-01-24T17:02:00.568 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_max_range ... ok 2023-01-24T17:02:00.619 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_positive_range ... ok 2023-01-24T17:02:00.678 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put ... ok 2023-01-24T17:02:00.771 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put_write_access ... ok 2023-01-24T17:02:00.820 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_put_authenticated ... ok 2023-01-24T17:02:00.868 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_raw_put_authenticated_expired ... ok 2023-01-24T17:02:00.882 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_one ... ok 2023-01-24T17:02:00.898 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_two ... ok 2023-01-24T17:02:00.912 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_ip ... ok 2023-01-24T17:02:00.929 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_dns_underscore ... ok 2023-01-24T17:02:00.969 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists ... ok 2023-01-24T17:02:01.006 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_get_location ... ok 2023-01-24T17:02:01.046 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists_nonowner ... ok 2023-01-24T17:02:01.085 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_overwrite_acl ... ok 2023-01-24T17:02:01.123 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_new_acl ... ok 2023-01-24T17:02:01.162 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_default ... ok 2023-01-24T17:02:01.194 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_during_create ... ok 2023-01-24T17:02:01.242 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned ... ok 2023-01-24T17:02:01.277 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_publicreadwrite ... ok 2023-01-24T17:02:01.309 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_authenticatedread ... ok 2023-01-24T17:02:01.367 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_acl_default ... ok 2023-01-24T17:02:01.420 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_during_create ... ok 2023-01-24T17:02:01.483 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned ... ok 2023-01-24T17:02:01.535 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_publicreadwrite ... ok 2023-01-24T17:02:01.589 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_authenticatedread ... ok 2023-01-24T17:02:01.656 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerread ... ok 2023-01-24T17:02:01.728 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerfullcontrol ... ok 2023-01-24T17:02:01.798 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_owner ... ok 2023-01-24T17:02:01.875 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_attributes ... ok 2023-01-24T17:02:01.916 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_private_to_private ... ok 2023-01-24T17:02:01.986 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_acl ... ok 2023-01-24T17:02:02.104 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_acl_write ... ok 2023-01-24T17:02:02.223 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_acl_writeacp ... ok 2023-01-24T17:02:02.347 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_acl_read ... ok 2023-01-24T17:02:02.467 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_acl_readacp ... ok 2023-01-24T17:02:02.617 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_fullcontrol ... ok 2023-01-24T17:02:02.712 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_read ... ok 2023-01-24T17:02:02.804 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_readacp ... ok 2023-01-24T17:02:02.916 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_write ... ok 2023-01-24T17:02:04.856 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_writeacp ... ok 2023-01-24T17:02:04.904 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_nonexist_user ... ok 2023-01-24T17:02:05.001 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_no_grants ... ok 2023-01-24T17:02:05.118 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_header_acl_grants ... ok 2023-01-24T17:02:05.181 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_header_acl_grants ... ok 2023-01-24T17:02:05.235 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email ... ok 2023-01-24T17:02:05.283 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email_not_exist ... ok 2023-01-24T17:02:05.366 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_revoke_all ... ok 2023-01-24T17:02:05.499 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_private ... ok 2023-01-24T17:02:05.742 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_private ... ok 2023-01-24T17:02:05.903 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicread ... ok 2023-01-24T17:02:06.031 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicread ... ok 2023-01-24T17:02:06.154 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicreadwrite ... ok 2023-01-24T17:02:06.273 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicreadwrite ... ok 2023-01-24T17:02:06.436 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_private ... ok 2023-01-24T17:02:06.572 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicread ... ok 2023-01-24T17:02:06.722 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicreadwrite ... ok 2023-01-24T17:02:06.918 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_private ... ok 2023-01-24T17:02:07.117 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicread ... ok 2023-01-24T17:02:07.336 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicreadwrite ... ok 2023-01-24T17:02:07.725 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_buckets_create_then_list ... ok 2023-01-24T17:02:08.112 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_buckets_list_ctime ... ok 2023-01-24T17:02:08.118 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_anonymous ... ok 2023-01-24T17:02:08.131 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_invalid_auth ... ok 2023-01-24T17:02:08.142 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_bad_auth ... ok 2023-01-24T17:02:08.326 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_alpha ... ok 2023-01-24T17:02:08.515 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_digit ... ok 2023-01-24T17:02:08.548 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_period ... ok 2023-01-24T17:02:08.581 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_hyphen ... ok 2023-01-24T17:02:08.682 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_not_overriding ... ok 2023-01-24T17:02:09.313 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_special_key_names ... ok 2023-01-24T17:02:09.441 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_special_prefix ... ok 2023-01-24T17:02:09.693 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_copy_zero_size ... ok 2023-01-24T17:02:09.814 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_copy_same_bucket ... ok 2023-01-24T17:02:09.891 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_copy_verify_contenttype ... ok 2023-01-24T17:02:09.959 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself ... ok 2023-01-24T17:02:10.041 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself_with_metadata ... ok 2023-01-24T17:02:10.319 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_copy_diff_bucket ... ok 2023-01-24T17:02:10.400 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_bucket ... ok 2023-01-24T17:02:10.699 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_object_bucket ... ok 2023-01-24T17:02:10.797 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_copy_canned_acl ... ok 2023-01-24T17:02:10.965 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_copy_retaining_metadata ... ok 2023-01-24T17:02:11.145 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_copy_replacing_metadata ... ok 2023-01-24T17:02:11.206 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_copy_bucket_not_found ... ok 2023-01-24T17:02:11.303 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_copy_key_not_found ... ok 2023-01-24T17:02:11.753 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_bucket ... ok 2023-01-24T17:02:11.910 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_url_encoding ... ok 2023-01-24T17:02:13.292 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioning_multipart_upload ... ok 2023-01-24T17:02:13.350 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_empty ... ok 2023-01-24T17:02:13.451 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_small ... ok 2023-01-24T17:02:13.761 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_small ... ok 2023-01-24T17:02:14.808 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_invalid_range ... ok 2023-01-24T17:02:14.982 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_without_range ... ok 2023-01-24T17:02:16.004 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_special_names ... ok 2023-01-24T17:02:17.080 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload ... ok 2023-01-24T17:02:19.611 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_versioned ... ok 2023-01-24T17:02:25.919 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_resend_part ... ok 2023-01-24T17:02:27.204 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_multiple_sizes ... ok 2023-01-24T17:02:28.745 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_multiple_sizes ... ok 2023-01-24T17:02:28.979 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_size_too_small ... ok 2023-01-24T17:02:29.486 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_contents ... ok 2023-01-24T17:02:29.908 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_overwrite_existing_object ... ok 2023-01-24T17:02:30.171 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload ... ok 2023-01-24T17:02:30.231 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload_not_found ... ok 2023-01-24T17:02:30.704 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload ... ok 2023-01-24T17:02:30.818 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload_owner ... ok 2023-01-24T17:02:30.979 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_missing_part ... ok 2023-01-24T17:02:31.063 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_incorrect_etag ... ok 2023-01-24T17:02:31.102 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_100_continue ... ok 2023-01-24T17:02:31.163 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_set_cors ... ok 2023-01-24T17:02:34.307 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_response ... ok 2023-01-24T17:02:37.374 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_wildcard ... ok 2023-01-24T17:02:40.427 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_cors_header_option ... ok 2023-01-24T17:02:40.496 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_set_bucket_tagging ... ok 2023-01-24T17:02:40.667 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_1mb ... ok 2023-01-24T17:02:40.931 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_4mb ... ok 2023-01-24T17:02:41.347 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_8mb ... ok 2023-01-24T17:02:41.600 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_1mb ... ok 2023-01-24T17:02:41.962 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_4mb ... ok 2023-01-24T17:02:42.617 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_8mb ... ok 2023-01-24T17:02:42.840 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_1mb ... ok 2023-01-24T17:02:43.240 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_4mb ... ok 2023-01-24T17:02:43.877 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_8mb ... ok 2023-01-24T17:02:44.083 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_atomic_conditional_write_1mb ... ok 2023-01-24T17:02:44.218 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_atomic_multipart_upload_write ... ok 2023-01-24T17:02:44.468 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_resend_first_finishes_last ... ok 2023-01-24T17:02:44.529 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_response_code ... ok 2023-01-24T17:02:48.238 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_ranged_big_request_response_code ... ok 2023-01-24T17:02:48.294 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_skip_leading_bytes_response_code ... ok 2023-01-24T17:02:48.349 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_return_trailing_bytes_response_code ... ok 2023-01-24T17:02:48.404 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_invalid_range ... ok 2023-01-24T17:02:48.454 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_empty_object ... ok 2023-01-24T17:02:48.558 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_create_suspend ... ok 2023-01-24T17:02:52.583 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove ... ok 2023-01-24T17:02:53.037 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove_head ... ok 2023-01-24T17:02:53.154 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_removal ... ok 2023-01-24T17:02:53.483 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite ... ok 2023-01-24T17:02:53.730 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite_suspended ... ok 2023-01-24T17:02:55.242 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_suspend_versions ... ok 2023-01-24T17:02:56.989 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_all ... ok 2023-01-24T17:03:03.969 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_special_names ... ok 2023-01-24T17:03:05.902 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_overwrite_multipart ... ok 2023-01-24T17:03:06.385 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_list_marker ... ok 2023-01-24T17:03:06.886 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_copy_obj_version ... ok 2023-01-24T17:03:07.112 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete ... ok 2023-01-24T17:03:07.472 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker ... ok 2023-01-24T17:03:07.563 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker_create ... ok 2023-01-24T17:03:07.828 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl ... ok 2023-01-24T17:03:07.983 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl_no_version_specified ... ok 2023-01-24T17:03:08.512 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioned_concurrent_object_create_and_remove ... ok 2023-01-24T17:03:08.567 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set ... ok 2023-01-24T17:03:08.664 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get ... ok 2023-01-24T17:03:08.719 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get_no_id ... ok 2023-01-24T17:03:08.760 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_id_too_long ... ok 2023-01-24T17:03:08.804 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_same_id ... ok 2023-01-24T17:03:08.857 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_invalid_status ... ok 2023-01-24T17:03:08.907 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_date ... ok 2023-01-24T17:03:08.952 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_invalid_date ... ok 2023-01-24T17:03:09.046 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent ... ok 2023-01-24T17:03:09.101 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_deletemarker ... ok 2023-01-24T17:03:09.151 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_filter ... ok 2023-01-24T17:03:09.205 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_empty_filter ... ok 2023-01-24T17:03:09.265 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_multipart ... ok 2023-01-24T17:03:09.315 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_set_invalid_date ... ok 2023-01-24T17:03:09.316 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition ... SKIP 2023-01-24T17:03:09.317 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_transition_single_rule_multi_trans ... SKIP 2023-01-24T17:03:09.317 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent_transition ... SKIP 2023-01-24T17:03:09.317 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition ... SKIP 2023-01-24T17:03:09.317 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_multiple_transition ... SKIP 2023-01-24T17:03:09.317 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_cloud_transition_large_obj ... SKIP 2023-01-24T17:03:09.377 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1b ... ok 2023-01-24T17:03:09.446 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1kb ... ok 2023-01-24T17:03:09.532 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1MB ... ok 2023-01-24T17:03:09.600 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_13b ... ok 2023-01-24T17:03:09.673 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_method_head ... ok 2023-01-24T17:03:09.731 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_present ... ok 2023-01-24T17:03:09.801 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_other_key ... ok 2023-01-24T17:03:09.842 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_invalid_md5 ... ok 2023-01-24T17:03:09.886 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_md5 ... ok 2023-01-24T17:03:09.933 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_key ... ok 2023-01-24T17:03:09.980 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_encryption_key_no_sse_c ... ok 2023-01-24T17:03:11.214 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_upload ... ok 2023-01-24T17:03:12.028 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_bad_download ... ok 2023-01-24T17:03:12.092 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_post_object_authenticated_request ... ok 2023-01-24T17:03:12.293 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_method_head ... ok 2023-01-24T17:03:12.379 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_present ... ok 2023-01-24T17:03:12.426 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_no_key ... ok 2023-01-24T17:03:12.467 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_not_declared ... ok 2023-01-24T17:03:14.353 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_upload ... ok 2023-01-24T17:03:15.167 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_1 ... ok 2023-01-24T17:03:15.989 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_2 ... ok 2023-01-24T17:03:16.095 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_post_object_authenticated_request ... ok 2023-01-24T17:03:16.184 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1b ... ok 2023-01-24T17:03:16.280 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1kb ... ok 2023-01-24T17:03:16.398 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1MB ... ok 2023-01-24T17:03:16.496 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_13b ... ok 2023-01-24T17:03:16.561 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_read_declare ... ok 2023-01-24T17:03:16.629 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy ... ok 2023-01-24T17:03:16.702 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy ... ok 2023-01-24T17:03:16.793 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_acl ... ok 2023-01-24T17:03:16.886 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_acl ... ok 2023-01-24T17:03:17.026 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_another_bucket ... ok 2023-01-24T17:03:17.205 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_another_bucket ... ok 2023-01-24T17:03:17.392 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_obj_tagging ... ok 2023-01-24T17:03:17.580 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_obj_head_tagging ... ok 2023-01-24T17:03:17.758 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_max_tags ... ok 2023-01-24T17:03:17.925 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_excess_tags ... ok 2023-01-24T17:03:18.111 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_max_kvsize_tags ... ok 2023-01-24T17:03:18.287 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_excess_key_tags ... ok 2023-01-24T17:03:18.470 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_excess_val_tags ... ok 2023-01-24T17:03:18.690 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_modify_tags ... ok 2023-01-24T17:03:18.891 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_delete_tags ... ok 2023-01-24T17:03:18.952 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_anonymous_request ... ok 2023-01-24T17:03:19.014 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_authenticated_request ... ok 2023-01-24T17:03:19.078 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_obj_with_tags ... ok 2023-01-24T17:03:19.272 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_tags_acl_public ... ok 2023-01-24T17:03:19.463 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_tags_acl_public ... ok 2023-01-24T17:03:19.674 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_delete_tags_obj_public ... ok 2023-01-24T17:03:19.871 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_atomic_upload_return_version_id ... ok 2023-01-24T17:03:21.971 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_multipart_upload_return_version_id ... ok 2023-01-24T17:03:22.111 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_existing_tag ... ok 2023-01-24T17:03:22.264 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_tagging_existing_tag ... ok 2023-01-24T17:03:22.455 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_tagging_existing_tag ... ok 2023-01-24T17:03:22.691 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source ... ok 2023-01-24T17:03:22.876 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source_meta ... ok 2023-01-24T17:03:22.939 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_acl ... ok 2023-01-24T17:03:23.072 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_grant ... ok 2023-01-24T17:03:23.120 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_c_s3 ... ok 2023-01-24T17:03:23.167 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_c_kms ... ok 2023-01-24T17:03:23.216 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_s3_kms ... ok 2023-01-24T17:03:23.266 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_obj_enc_conflict_bad_enc_kms ... ok 2023-01-24T17:03:23.340 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_noenc ... ok 2023-01-24T17:03:23.393 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_kms_s3 ... ok 2023-01-24T17:03:23.540 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_acl_existing_tag ... ok 2023-01-24T17:03:23.639 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock ... ok 2023-01-24T17:03:23.689 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_bucket ... ok 2023-01-24T17:03:23.730 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_with_days_and_years ... ok 2023-01-24T17:03:23.766 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_days ... ok 2023-01-24T17:03:23.805 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_years ... ok 2023-01-24T17:03:23.846 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_mode ... ok 2023-01-24T17:03:23.887 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_status ... ok 2023-01-24T17:03:23.924 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_suspend_versioning ... ok 2023-01-24T17:03:23.974 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock ... ok 2023-01-24T17:03:24.013 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock_invalid_bucket ... ok 2023-01-24T17:03:24.115 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention ... ok 2023-01-24T17:03:24.180 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_bucket ... ok 2023-01-24T17:03:24.254 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_mode ... ok 2023-01-24T17:03:24.414 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention ... ok 2023-01-24T17:03:24.524 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_iso8601 ... ok 2023-01-24T17:03:24.622 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_invalid_bucket ... ok 2023-01-24T17:03:24.805 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_versionid ... ok 2023-01-24T17:03:24.974 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_override_default_retention ... ok 2023-01-24T17:03:25.145 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_increase_period ... ok 2023-01-24T17:03:25.294 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period ... ok 2023-01-24T17:03:25.472 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period_bypass ... ok 2023-01-24T17:03:25.660 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention ... ok 2023-01-24T17:03:25.841 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention_and_marker ... ok 2023-01-24T17:03:26.022 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_multi_delete_object_with_retention ... ok 2023-01-24T17:03:26.112 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold ... ok 2023-01-24T17:03:26.168 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_bucket ... ok 2023-01-24T17:03:26.230 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_status ... ok 2023-01-24T17:03:26.373 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold ... ok 2023-01-24T17:03:26.477 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold_invalid_bucket ... ok 2023-01-24T17:03:26.619 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_on ... ok 2023-01-24T17:03:26.717 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_off ... ok 2023-01-24T17:03:26.855 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_metadata ... ok 2023-01-24T17:03:26.979 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_uploading_obj ... ok 2023-01-24T17:03:27.050 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_governance_with_bypass ... ok 2023-01-24T17:03:27.121 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_governance_without_bypass ... ok 2023-01-24T17:03:27.189 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_object_lock_changing_mode_from_compliance ... ok 2023-01-24T17:03:27.264 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifmatch_good ... ok 2023-01-24T17:03:27.344 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_copy_object_ifnonematch_failed ... ok 2023-01-24T17:03:27.375 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_user_policy ... ok 2023-01-24T17:03:27.422 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_policy_status ... ok 2023-01-24T17:03:27.476 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_public_acl_bucket_policy_status ... ok 2023-01-24T17:03:27.528 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_authpublic_acl_bucket_policy_status ... ok 2023-01-24T17:03:27.589 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_publicpolicy_acl_bucket_policy_status ... ok 2023-01-24T17:03:27.646 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_acl_bucket_policy_status ... ok 2023-01-24T17:03:27.699 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_deny_bucket_policy_status ... ok 2023-01-24T17:03:27.741 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_default_public_block ... ok 2023-01-24T17:03:27.797 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_public_block ... ok 2023-01-24T17:03:27.863 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_block_public_put_bucket_acls ... ok 2023-01-24T17:03:27.926 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_block_public_object_canned_acls ... ok 2023-01-24T17:03:27.986 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_block_public_policy ... ok 2023-01-24T17:03:28.094 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_ignore_public_acls ... ok 2023-01-24T17:03:29.208 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_on_a_bucket_with_policy ... ok 2023-01-24T17:03:29.293 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_put_bucket_encryption_kms ... ok 2023-01-24T17:03:29.345 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_encryption_kms ... ok 2023-01-24T17:03:29.457 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_delete_bucket_encryption_kms ... ok 2023-01-24T17:03:29.557 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_default_post_object_authenticated_request ... ok 2023-01-24T17:03:30.582 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_generate_where_clause ... ok 2023-01-24T17:03:31.620 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_generate_projection ... ok 2023-01-24T17:03:31.682 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_count_operation ... ok 2023-01-24T17:03:32.210 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_column_sum_min_max ... ok 2023-01-24T17:03:32.915 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_nullif_expressions ... ok 2023-01-24T17:03:33.202 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_nulliftrue_expressions ... ok 2023-01-24T17:03:33.423 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_is_not_null_expressions ... ok 2023-01-24T17:03:33.496 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_lowerupper_expressions ... ok 2023-01-24T17:03:34.171 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_in_expressions ... ok 2023-01-24T17:03:34.640 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_true_false_in_expressions ... ok 2023-01-24T17:03:35.362 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_like_expressions ... ok 2023-01-24T17:03:35.755 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_truefalselike_expressions ... ok 2023-01-24T17:03:36.028 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_complex_expressions ... ok 2023-01-24T17:03:36.233 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_alias ... ok 2023-01-24T17:03:36.399 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_alias_cyclic_refernce ... ok 2023-01-24T17:03:36.652 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_datetime ... ok 2023-01-24T17:03:37.005 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_true_false_datetime ... ok 2023-01-24T17:03:37.120 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_csv_parser ... ok 2023-01-24T17:03:37.335 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_csv_definition ... ok 2023-01-24T17:03:37.566 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_schema_definition ... ok 2023-01-24T17:03:37.812 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_when_then_else_expressions ... ok 2023-01-24T17:03:38.084 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_coalesce_expressions ... ok 2023-01-24T17:03:38.314 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_cast_expressions ... ok 2023-01-24T17:03:38.315 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_version ... ok 2023-01-24T17:03:38.764 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_trim_expressions ... ok 2023-01-24T17:03:39.212 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_truefalse_trim_expressions ... ok 2023-01-24T17:03:39.485 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_escape_expressions ... ok 2023-01-24T17:03:39.717 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_case_value_expressions ... ok 2023-01-24T17:03:39.892 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_bool_cast_expressions ... ok 2023-01-24T17:03:49.763 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_progress_expressions ... ok 2023-01-24T17:03:49.763 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_s3select.test_output_serial_expressions ... ok 2023-01-24T17:05:12.496 INFO:teuthology.orchestra.run.smithi103.stderr:s3tests_boto3.functional.test_utils.test_generate ... ok 2023-01-24T17:05:12.496 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-01-24T17:05:12.496 INFO:teuthology.orchestra.run.smithi103.stderr:---------------------------------------------------------------------- 2023-01-24T17:05:12.497 INFO:teuthology.orchestra.run.smithi103.stderr:Ran 619 tests in 256.985s 2023-01-24T17:05:12.497 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-01-24T17:05:12.497 INFO:teuthology.orchestra.run.smithi103.stderr:OK (SKIP=79) 2023-01-24T17:05:12.524 INFO:teuthology.orchestra.run.smithi103.stderr:Exception ignored in: .generate at 0x7f8543e98cf0> 2023-01-24T17:05:12.525 INFO:teuthology.orchestra.run.smithi103.stderr:RuntimeError: generator ignored GeneratorExit 2023-01-24T17:05:12.608 DEBUG:tasks.s3tests:Scanning radosgw logs for leaked encryption keys... 2023-01-24T17:05:12.609 DEBUG:teuthology.orchestra.run.smithi103:> grep --binary-files=text pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= /var/log/ceph/rgw.ceph.client.0.log 2023-01-24T17:05:12.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T17:05:12.675 INFO:tasks.s3tests:Cleaning up boto... 2023-01-24T17:05:12.675 DEBUG:teuthology.orchestra.run.smithi103:> rm /home/ubuntu/cephtest/boto.cfg 2023-01-24T17:05:12.682 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2023-01-24T17:05:12.718 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:05:12.718 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:05:12.729 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.720+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.724+0000 7f51d5404e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:12.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.724+0000 7f51d5404e40 20 realm 2023-01-24T17:05:12.732 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.724+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.724+0000 7f51d5404e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:12.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.724+0000 7f51d5404e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:05:12.733 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.724+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.724+0000 7f51d5404e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:12.734 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.724+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.724+0000 7f51d5404e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:05:12.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.724+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.728+0000 7f51d5404e40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:05:12.735 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.728+0000 7f51d5404e40 20 searching for the correct realm 2023-01-24T17:05:12.755 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.748+0000 7f51d5404e40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:05:12.755 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.748+0000 7f51d5404e40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:05:12.755 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.748+0000 7f51d5404e40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:05:12.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.748+0000 7f51d5404e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:05:12.756 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.748+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.748+0000 7f51d5404e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:12.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.748+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.748+0000 7f51d5404e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:05:12.757 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.748+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.748+0000 7f51d5404e40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:05:12.758 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.748+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.759 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.748+0000 7f51d5404e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:12.759 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.748+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.759 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.748+0000 7f51d5404e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:12.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.748+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.760 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.752+0000 7f51d5404e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:05:12.778 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:05:12.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:05:12.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:05:12.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:05:12.779 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.780 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:05:12.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:05:12.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:05:12.781 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.782 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:05:12.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:12.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 20 zone default found 2023-01-24T17:05:12.783 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 4 Realm: () 2023-01-24T17:05:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 4 ZoneGroup: default (0d4b5768-74c8-4719-bc1b-4550aa389781) 2023-01-24T17:05:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.772+0000 7f51d5404e40 4 Zone: default (2ee3967b-537d-4521-a85e-a66f0a1f41f2) 2023-01-24T17:05:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.776+0000 7f51d5404e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:05:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.776+0000 7f51d5404e40 20 zonegroup default 2023-01-24T17:05:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.776+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.776+0000 7f51d5404e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:12.785 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.776+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.776+0000 7f51d5404e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:12.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.776+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.776+0000 7f51d5404e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:12.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.776+0000 7f51d5404e40 20 started sync module instance, tier type = 2023-01-24T17:05:12.786 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.776+0000 7f51d5404e40 20 started zone id=2ee3967b-537d-4521-a85e-a66f0a1f41f2 (name=default) with tier type = 2023-01-24T17:05:12.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.804+0000 7f51d5404e40 20 add_watcher() i=0 2023-01-24T17:05:12.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.804+0000 7f51d5404e40 20 add_watcher() i=1 2023-01-24T17:05:12.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.804+0000 7f51d5404e40 20 add_watcher() i=2 2023-01-24T17:05:12.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.804+0000 7f51d5404e40 20 add_watcher() i=3 2023-01-24T17:05:12.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.804+0000 7f51d5404e40 20 add_watcher() i=4 2023-01-24T17:05:12.813 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.804+0000 7f51d5404e40 20 add_watcher() i=5 2023-01-24T17:05:12.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.804+0000 7f51d5404e40 20 add_watcher() i=6 2023-01-24T17:05:12.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.808+0000 7f51d5404e40 20 add_watcher() i=7 2023-01-24T17:05:12.814 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.808+0000 7f51d5404e40 2 all 8 watchers are set, enabling cache 2023-01-24T17:05:12.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.808+0000 7f51a3fe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:05:12.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.808+0000 7f51d5404e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:05:12.818 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.808+0000 7f51d5404e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:05:12.913 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.904+0000 7f5112ffd700 20 reqs_thread_entry: start 2023-01-24T17:05:12.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.904+0000 7f5110ff9700 20 reqs_thread_entry: start 2023-01-24T17:05:12.914 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.904+0000 7f51d5404e40 20 init_complete bucket index max shards: 11 2023-01-24T17:05:12.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.908+0000 7f51d5404e40 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:05:12.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.908+0000 7f51067fc700 20 reqs_thread_entry: start 2023-01-24T17:05:12.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.908+0000 7f51d5404e40 20 get_system_obj_state: rctx=0x7ffe2833b048 obj=default.rgw.meta:users.uid:foo.client.0 state=0x55d4c489fef0 s->prefetch_data=0 2023-01-24T17:05:12.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.908+0000 7f51d5404e40 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2023-01-24T17:05:12.915 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.908+0000 7f5106ffd700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:05:43 2023 2023-01-24T17:05:12.916 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-01-24T17:05:12.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.908+0000 7f51d5404e40 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2023-01-24T17:05:12.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.908+0000 7f51d5404e40 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:05:12.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.908+0000 7f51d5404e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:05:12.917 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.908+0000 7f51d5404e40 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:05:12.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.908+0000 7f51d5404e40 20 Read xattr: user.rgw.idtag 2023-01-24T17:05:12.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.908+0000 7f51d5404e40 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:05:12.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.908+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.908+0000 7f51d5404e40 20 rados_obj.operate() r=0 bl.length=469 2023-01-24T17:05:12.918 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.908+0000 7f51d5404e40 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x13 2023-01-24T17:05:12.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.908+0000 7f51d5404e40 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2023-01-24T17:05:12.919 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.908+0000 7f51d5404e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:05:12.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 20 get_system_obj_state: rctx=0x7ffe2833acf0 obj=default.rgw.meta:root:.bucket.meta.test:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518 state=0x55d4c489fef0 s->prefetch_data=0 2023-01-24T17:05:12.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.test:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518 : miss 2023-01-24T17:05:12.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.test:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518 info.flags=0x16 2023-01-24T17:05:12.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 10 adding default.rgw.meta+root+.bucket.meta.test:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518 to cache LRU end 2023-01-24T17:05:12.920 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:05:12.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:05:12.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:05:12.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 20 Read xattr: user.rgw.acl 2023-01-24T17:05:12.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 20 Read xattr: user.rgw.idtag 2023-01-24T17:05:12.921 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.test:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:05:12.922 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.923 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 20 rados_obj.operate() r=0 bl.length=364 2023-01-24T17:05:12.923 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.test:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518 info.flags=0x13 2023-01-24T17:05:12.923 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 10 moving default.rgw.meta+root+.bucket.meta.test:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518 to cache LRU end 2023-01-24T17:05:12.923 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:05:12.924 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:05:12.924 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518 2023-01-24T17:05:12.924 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :test[2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-01-24T17:05:12.924 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 09AB9F 2023-01-24T17:05:12.924 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 10 cls_bucket_list_unordered: got A04652 2023-01-24T17:05:12.925 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.912+0000 7f51d5404e40 10 cls_bucket_list_unordered: got B9AC0B 2023-01-24T17:05:12.925 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.916+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 96A700 2023-01-24T17:05:12.925 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.916+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 1D5B99 2023-01-24T17:05:12.925 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.916+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 7976D4 2023-01-24T17:05:12.925 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.916+0000 7f51d5404e40 10 cls_bucket_list_unordered: got E8C8CE 2023-01-24T17:05:12.926 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.916+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 861D54 2023-01-24T17:05:12.926 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.916+0000 7f51d5404e40 10 cls_bucket_list_unordered: got A6F2F9 2023-01-24T17:05:12.927 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.916+0000 7f51d5404e40 10 cls_bucket_list_unordered: got DDB462 2023-01-24T17:05:12.927 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.916+0000 7f51d5404e40 10 cls_bucket_list_unordered: got E84474 2023-01-24T17:05:12.927 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.916+0000 7f51d5404e40 10 cls_bucket_list_unordered: got BAC2AD 2023-01-24T17:05:12.927 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.916+0000 7f51d5404e40 10 cls_bucket_list_unordered: got CDCE6C 2023-01-24T17:05:12.927 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.920+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 2576DF 2023-01-24T17:05:12.928 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.920+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 3CABB1 2023-01-24T17:05:12.928 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.920+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 433E1D 2023-01-24T17:05:12.928 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.920+0000 7f51d5404e40 10 cls_bucket_list_unordered: got ACE6F3 2023-01-24T17:05:12.928 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.920+0000 7f51d5404e40 10 cls_bucket_list_unordered: got E31790 2023-01-24T17:05:12.928 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.920+0000 7f51d5404e40 10 cls_bucket_list_unordered: got E8AAA8 2023-01-24T17:05:12.929 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.920+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 97E66E 2023-01-24T17:05:12.929 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.920+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 75D6A8 2023-01-24T17:05:12.930 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.920+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 7BDE23 2023-01-24T17:05:12.930 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.920+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 34F93C 2023-01-24T17:05:12.930 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.920+0000 7f51d5404e40 10 cls_bucket_list_unordered: got E95BF1 2023-01-24T17:05:12.931 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.924+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 186545 2023-01-24T17:05:12.931 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.924+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 29A74F 2023-01-24T17:05:12.931 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.924+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 4405D1 2023-01-24T17:05:12.932 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.924+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 10789E 2023-01-24T17:05:12.932 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.924+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 9D454A 2023-01-24T17:05:12.932 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.924+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:09AB9F state=0x55d4c4a96cf8 s->prefetch_data=0 2023-01-24T17:05:12.932 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.924+0000 7f51d5404e40 10 manifest: total_size = 399311 2023-01-24T17:05:12.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.924+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.366740871658857192 2023-01-24T17:05:12.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.924+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:09AB9F state=0x55d4c4a96cf8 s->prefetch_data=0 2023-01-24T17:05:12.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.924+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96cf8 2023-01-24T17:05:12.933 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.924+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.0 2023-01-24T17:05:12.937 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.928+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : miss 2023-01-24T17:05:12.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.928+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.938 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:12.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 10 cache put: name=default.rgw.log++bucket.sync-source-hints.test info.flags=0x0 2023-01-24T17:05:12.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 10 adding default.rgw.log++bucket.sync-source-hints.test to cache LRU end 2023-01-24T17:05:12.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : miss 2023-01-24T17:05:12.939 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 10 cache put: name=default.rgw.log++bucket.sync-target-hints.test info.flags=0x0 2023-01-24T17:05:12.940 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 10 adding default.rgw.log++bucket.sync-target-hints.test to cache LRU end 2023-01-24T17:05:12.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:12.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:12.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:12.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:A04652 state=0x55d4c4a96cb8 s->prefetch_data=0 2023-01-24T17:05:12.941 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 10 manifest: total_size = 575598 2023-01-24T17:05:12.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.2221965802296532615 2023-01-24T17:05:12.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:A04652 state=0x55d4c4a96cb8 s->prefetch_data=0 2023-01-24T17:05:12.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96cb8 2023-01-24T17:05:12.942 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.932+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.0 2023-01-24T17:05:12.945 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.936+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:12.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.936+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:12.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.936+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:12.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.936+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:12.946 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.936+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:12.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.936+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:B9AC0B state=0x55d4c4a96cb8 s->prefetch_data=0 2023-01-24T17:05:12.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.940+0000 7f51d5404e40 10 manifest: total_size = 180000 2023-01-24T17:05:12.947 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.940+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.9545499709742689989 2023-01-24T17:05:12.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.940+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:B9AC0B state=0x55d4c4a96cb8 s->prefetch_data=0 2023-01-24T17:05:12.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.940+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96cb8 2023-01-24T17:05:12.948 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.940+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.0 2023-01-24T17:05:12.953 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.944+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:12.953 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.944+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:12.954 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.944+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:12.954 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.944+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:12.954 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.944+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:12.954 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.944+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:96A700 state=0x55d4c4a96cb8 s->prefetch_data=0 2023-01-24T17:05:12.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.948+0000 7f51d5404e40 10 manifest: total_size = 399155 2023-01-24T17:05:12.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.948+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.1337325829296317726 2023-01-24T17:05:12.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.948+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:96A700 state=0x55d4c4a96cb8 s->prefetch_data=0 2023-01-24T17:05:12.955 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.948+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96cb8 2023-01-24T17:05:12.956 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.948+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.1 2023-01-24T17:05:12.959 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.952+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:12.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.952+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:12.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.952+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:12.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.952+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:12.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.952+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:12.960 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.952+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:1D5B99 state=0x55d4c4a96cb8 s->prefetch_data=0 2023-01-24T17:05:12.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.952+0000 7f51d5404e40 10 manifest: total_size = 399183 2023-01-24T17:05:12.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.952+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.14416851808473279634 2023-01-24T17:05:12.961 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.952+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:1D5B99 state=0x55d4c4a96cb8 s->prefetch_data=0 2023-01-24T17:05:12.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.952+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96cb8 2023-01-24T17:05:12.962 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.952+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.2 2023-01-24T17:05:12.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.960+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:12.967 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.960+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:12.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.960+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:12.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.960+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:12.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.960+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:12.968 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.960+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:7976D4 state=0x55d4c4a96cb8 s->prefetch_data=0 2023-01-24T17:05:12.969 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.960+0000 7f51d5404e40 10 manifest: total_size = 399303 2023-01-24T17:05:12.969 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.960+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.5032975628553826475 2023-01-24T17:05:12.969 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.960+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:7976D4 state=0x55d4c4a96cb8 s->prefetch_data=0 2023-01-24T17:05:12.970 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.960+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96cb8 2023-01-24T17:05:12.970 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.960+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.2 2023-01-24T17:05:12.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.968+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:12.974 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.968+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:12.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.968+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:12.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.968+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:12.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.968+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:12.975 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.968+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:E8C8CE state=0x55d4c4a96cb8 s->prefetch_data=0 2023-01-24T17:05:12.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.968+0000 7f51d5404e40 10 manifest: total_size = 399032 2023-01-24T17:05:12.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.968+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.1951360313882652057 2023-01-24T17:05:12.976 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.968+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:E8C8CE state=0x55d4c4a96cb8 s->prefetch_data=0 2023-01-24T17:05:12.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.968+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96cb8 2023-01-24T17:05:12.977 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.968+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.2 2023-01-24T17:05:12.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.976+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:12.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.976+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:12.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.976+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:12.982 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.976+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:12.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.976+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:12.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.976+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:861D54 state=0x55d4c4a96cb8 s->prefetch_data=0 2023-01-24T17:05:12.983 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.976+0000 7f51d5404e40 10 manifest: total_size = 399140 2023-01-24T17:05:12.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.976+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.15194685337479394628 2023-01-24T17:05:12.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.976+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:861D54 state=0x55d4c4a96cb8 s->prefetch_data=0 2023-01-24T17:05:12.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.976+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96cb8 2023-01-24T17:05:12.984 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.976+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.3 2023-01-24T17:05:12.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.980+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:12.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.980+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:12.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.980+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:12.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.980+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:12.989 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.980+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:12.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.980+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:A6F2F9 state=0x55d4c4a96c98 s->prefetch_data=0 2023-01-24T17:05:12.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.980+0000 7f51d5404e40 10 manifest: total_size = 399230 2023-01-24T17:05:12.990 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.980+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.1800936707597648763 2023-01-24T17:05:12.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.980+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:A6F2F9 state=0x55d4c4a96c98 s->prefetch_data=0 2023-01-24T17:05:12.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.980+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c98 2023-01-24T17:05:12.991 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.980+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.3 2023-01-24T17:05:12.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.988+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:12.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.988+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:12.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.988+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:12.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.988+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:12.996 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.988+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:12.997 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.988+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:DDB462 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:12.997 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.988+0000 7f51d5404e40 10 manifest: total_size = 399142 2023-01-24T17:05:12.997 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.988+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.15186849894311076052 2023-01-24T17:05:12.998 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.988+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:DDB462 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:12.998 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.988+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:12.999 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.988+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.3 2023-01-24T17:05:13.003 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.996+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.003 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.996+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.003 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.996+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.003 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.996+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.996+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.004 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.996+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:E84474 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.005 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.996+0000 7f51d5404e40 10 manifest: total_size = 399027 2023-01-24T17:05:13.005 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.996+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.7838055132309774296 2023-01-24T17:05:13.005 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.996+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:E84474 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.005 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.996+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.005 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:12.996+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.3 2023-01-24T17:05:13.011 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.004+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.011 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.004+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.011 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.004+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.011 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.004+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.012 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.004+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.012 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.004+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:BAC2AD state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.012 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.004+0000 7f51d5404e40 10 manifest: total_size = 643256 2023-01-24T17:05:13.013 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.004+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.11100868907294571618 2023-01-24T17:05:13.013 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.004+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:BAC2AD state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.013 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.004+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.013 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.004+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.4 2023-01-24T17:05:13.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.012+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.012+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.012+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.012+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.019 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.012+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.012+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:CDCE6C state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.012+0000 7f51d5404e40 10 manifest: total_size = 399039 2023-01-24T17:05:13.020 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.012+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.7017077473085072278 2023-01-24T17:05:13.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.012+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:CDCE6C state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.012+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.021 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.012+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.4 2023-01-24T17:05:13.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.016+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.016+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.016+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.026 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.016+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.016+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.027 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.020+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:2576DF state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.020+0000 7f51d5404e40 10 manifest: total_size = 117 2023-01-24T17:05:13.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.020+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.11332109039499706031 2023-01-24T17:05:13.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.020+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:2576DF state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.028 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.020+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.029 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.020+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.5 2023-01-24T17:05:13.031 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.024+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.024+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.024+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.024+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.024+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.032 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.024+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:3CABB1 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.024+0000 7f51d5404e40 10 manifest: total_size = 360305 2023-01-24T17:05:13.033 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.024+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.1809729134950386788 2023-01-24T17:05:13.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.024+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:3CABB1 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.024+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.034 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.024+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.5 2023-01-24T17:05:13.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.028+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.028+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.037 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.028+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.028+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.028+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.038 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.028+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:433E1D state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.039 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.028+0000 7f51d5404e40 10 manifest: total_size = 573896 2023-01-24T17:05:13.039 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.028+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.5576241615137711782 2023-01-24T17:05:13.039 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.028+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:433E1D state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.039 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.028+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.040 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.028+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.5 2023-01-24T17:05:13.043 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.036+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.043 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.036+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.043 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.036+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.044 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.036+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.044 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.036+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.044 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.036+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:ACE6F3 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.044 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.036+0000 7f51d5404e40 10 manifest: total_size = 39909894 2023-01-24T17:05:13.045 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.036+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.9810822982702005321 2023-01-24T17:05:13.045 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.036+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:ACE6F3 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.045 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.036+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.045 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.036+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.5 2023-01-24T17:05:13.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.050 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:05:13.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=0 2023-01-24T17:05:13.051 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:05:13.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=0 2023-01-24T17:05:13.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:05:13.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): result: ofs=16777216 stripe_ofs=16777216 part_ofs=0 rule->part_size=0 2023-01-24T17:05:13.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:05:13.052 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): result: ofs=20971520 stripe_ofs=20971520 part_ofs=0 rule->part_size=0 2023-01-24T17:05:13.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:05:13.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): result: ofs=25165824 stripe_ofs=25165824 part_ofs=0 rule->part_size=0 2023-01-24T17:05:13.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:05:13.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): result: ofs=29360128 stripe_ofs=29360128 part_ofs=0 rule->part_size=0 2023-01-24T17:05:13.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:05:13.053 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): result: ofs=33554432 stripe_ofs=33554432 part_ofs=0 rule->part_size=0 2023-01-24T17:05:13.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:05:13.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): result: ofs=37748736 stripe_ofs=37748736 part_ofs=0 rule->part_size=0 2023-01-24T17:05:13.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-01-24T17:05:13.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 RGWObjManifest::operator++(): result: ofs=39909894 stripe_ofs=39909894 part_ofs=0 rule->part_size=0 2023-01-24T17:05:13.054 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.040+0000 7f51d5404e40 20 garbage collection: RGWGC::send_chain - on object name: gc.11tag is: 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.9810822982702005321 2023-01-24T17:05:13.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.044+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:E31790 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.055 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.044+0000 7f51d5404e40 10 manifest: total_size = 590176 2023-01-24T17:05:13.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.044+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.2993895197569359518 2023-01-24T17:05:13.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.044+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:E31790 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.044+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.056 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.044+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.5 2023-01-24T17:05:13.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.048+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.048+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.048+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.048+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.058 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.048+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.048+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:E8AAA8 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.059 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.052+0000 7f51d5404e40 10 manifest: total_size = 399279 2023-01-24T17:05:13.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.052+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.17935582630485127384 2023-01-24T17:05:13.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.052+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:E8AAA8 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.052+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.060 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.052+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.5 2023-01-24T17:05:13.064 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.056+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.065 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.056+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.065 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.056+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.065 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.056+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.065 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.056+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.065 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.056+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:97E66E state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.056+0000 7f51d5404e40 10 manifest: total_size = 5 2023-01-24T17:05:13.066 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.056+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.16112306199381210996 2023-01-24T17:05:13.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.056+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:97E66E state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.056+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.067 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.056+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.6 2023-01-24T17:05:13.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.064+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.070 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.064+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.064+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.064+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.064+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.071 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.064+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:75D6A8 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.064+0000 7f51d5404e40 10 manifest: total_size = 399309 2023-01-24T17:05:13.072 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.064+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.335023168293782973 2023-01-24T17:05:13.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.064+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:75D6A8 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.064+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.073 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.064+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.7 2023-01-24T17:05:13.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.068+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.068+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.068+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.077 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.068+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.068+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.068+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:7BDE23 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.078 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.068+0000 7f51d5404e40 10 manifest: total_size = 180000 2023-01-24T17:05:13.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.068+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.4418370538064576454 2023-01-24T17:05:13.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.068+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:7BDE23 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.068+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.079 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.068+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.7 2023-01-24T17:05:13.084 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.076+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.084 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.076+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.085 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.076+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.085 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.076+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.085 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.076+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.085 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.076+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:34F93C state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.086 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.076+0000 7f51d5404e40 10 manifest: total_size = 5 2023-01-24T17:05:13.086 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.076+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.5726591371029636859 2023-01-24T17:05:13.086 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.076+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:34F93C state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.087 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.076+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.087 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.076+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.8 2023-01-24T17:05:13.091 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.084+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.091 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.084+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.091 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.084+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.092 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.084+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.092 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.084+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.092 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.084+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:E95BF1 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.093 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.084+0000 7f51d5404e40 10 manifest: total_size = 644720 2023-01-24T17:05:13.093 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.084+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.3826846402338018763 2023-01-24T17:05:13.093 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.084+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:E95BF1 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.094 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.084+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.094 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.084+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.8 2023-01-24T17:05:13.098 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.092+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.098 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.092+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.098 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.092+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.099 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.092+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.099 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.092+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.099 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.092+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:186545 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.100 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.092+0000 7f51d5404e40 10 manifest: total_size = 399272 2023-01-24T17:05:13.100 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.092+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.13256368593453981066 2023-01-24T17:05:13.100 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.092+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:186545 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.100 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.092+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.100 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.092+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.9 2023-01-24T17:05:13.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.096+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.096+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.104 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.096+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.096+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.096+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.105 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.096+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:29A74F state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.096+0000 7f51d5404e40 10 manifest: total_size = 41 2023-01-24T17:05:13.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.096+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.7922894519288734727 2023-01-24T17:05:13.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.096+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:29A74F state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.096+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.106 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.096+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.9 2023-01-24T17:05:13.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.104+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.112 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.104+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.104+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.104+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.104+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.113 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.104+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:4405D1 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.104+0000 7f51d5404e40 10 manifest: total_size = 399001 2023-01-24T17:05:13.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.104+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.1682155841879390011 2023-01-24T17:05:13.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.104+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:4405D1 state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.114 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.104+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.115 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.104+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.9 2023-01-24T17:05:13.120 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.112+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.112+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.112+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.112+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.121 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.112+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.112+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:10789E state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.112+0000 7f51d5404e40 10 manifest: total_size = 49195 2023-01-24T17:05:13.122 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.112+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.10591544298517463341 2023-01-24T17:05:13.123 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.112+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:10789E state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.123 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.112+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.123 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.112+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.10 2023-01-24T17:05:13.128 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.120+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.120+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.120+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.120+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.129 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.120+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.120+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:9D454A state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.120+0000 7f51d5404e40 10 manifest: total_size = 398971 2023-01-24T17:05:13.130 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.120+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.11964185852223033996 2023-01-24T17:05:13.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.120+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=test:9D454A state=0x55d4c4a96c28 s->prefetch_data=0 2023-01-24T17:05:13.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.120+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a96c28 2023-01-24T17:05:13.131 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.120+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518.10 2023-01-24T17:05:13.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.128+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test : hit (negative entry) 2023-01-24T17:05:13.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.128+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test : hit (negative entry) 2023-01-24T17:05:13.137 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.128+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.128+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.128+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.128+0000 7f51d5404e40 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-01-24T17:05:13.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.128+0000 7f51d5404e40 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :test[2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-01-24T17:05:13.138 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.128+0000 7f51d5404e40 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-01-24T17:05:13.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.132+0000 7f51d5404e40 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-01-24T17:05:13.141 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.132+0000 7f51d5404e40 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-01-24T17:05:13.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.132+0000 7f51d5404e40 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-01-24T17:05:13.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.132+0000 7f51d5404e40 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-01-24T17:05:13.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.132+0000 7f51d5404e40 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test[2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518])"; uploads.size()=0; is_truncated=0 2023-01-24T17:05:13.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.132+0000 7f51d5404e40 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:test[2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518]) 2023-01-24T17:05:13.142 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.132+0000 7f51d5404e40 20 lifecycle: get_system_obj_state: rctx=0x7ffe2833a7a8 obj=default.rgw.meta:root:test state=0x55d4c489fef0 s->prefetch_data=0 2023-01-24T17:05:13.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.132+0000 7f51d5404e40 10 lifecycle: cache get: name=default.rgw.meta+root+test : miss 2023-01-24T17:05:13.143 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51d5404e40 10 lifecycle: cache put: name=default.rgw.meta+root+test info.flags=0x16 2023-01-24T17:05:13.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51d5404e40 10 lifecycle: adding default.rgw.meta+root+test to cache LRU end 2023-01-24T17:05:13.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51d5404e40 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:05:13.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51d5404e40 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:05:13.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51d5404e40 20 lifecycle: Read xattr: user.rgw.idtag 2023-01-24T17:05:13.144 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51d5404e40 10 lifecycle: cache get: name=default.rgw.meta+root+test : type miss (requested=0x13, cached=0x16) 2023-01-24T17:05:13.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51d5404e40 20 lifecycle: rados->read ofs=0 len=0 2023-01-24T17:05:13.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51d5404e40 20 lifecycle: rados_obj.operate() r=0 bl.length=186 2023-01-24T17:05:13.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51d5404e40 10 lifecycle: cache put: name=default.rgw.meta+root+test info.flags=0x13 2023-01-24T17:05:13.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51d5404e40 10 lifecycle: moving default.rgw.meta+root+test to cache LRU end 2023-01-24T17:05:13.145 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51d5404e40 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:05:13.146 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51d5404e40 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.test:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518 info.flags=0x17 2023-01-24T17:05:13.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51d5404e40 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.test:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518 to cache LRU end 2023-01-24T17:05:13.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51d5404e40 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:05:13.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51d5404e40 10 lifecycle: distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.test:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518, ofs0, ns] 2023-01-24T17:05:13.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51cbfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562548 cookie 94372320285056 notifier 4247 bl.length()=800 2023-01-24T17:05:13.147 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51cbfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.test:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518 info.flags=0x17 2023-01-24T17:05:13.148 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51cbfff700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.test:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518 to cache LRU end 2023-01-24T17:05:13.148 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.136+0000 7f51cbfff700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:05:13.148 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.140+0000 7f51d5404e40 20 lifecycle: handle_bi_update(): bucket=:test[2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518]): orig_sources= new_sources= 2023-01-24T17:05:13.148 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.140+0000 7f51d5404e40 20 lifecycle: handle_bi_update(): bucket=:test[2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518]): potential sources added=[] removed=[] 2023-01-24T17:05:13.148 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.140+0000 7f51d5404e40 20 lifecycle: handle_bi_update(): bucket=:test[2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518]): orig_dests= new_dests= 2023-01-24T17:05:13.149 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.140+0000 7f51d5404e40 20 lifecycle: handle_bi_update(): bucket=:test[2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518]): potential dests added=[] removed=[] 2023-01-24T17:05:13.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.152+0000 7f51d5404e40 20 get_system_obj_state: rctx=0x7ffe2833a228 obj=default.rgw.meta:root:test state=0x55d4c489fef0 s->prefetch_data=0 2023-01-24T17:05:13.158 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.152+0000 7f51d5404e40 10 cache get: name=default.rgw.meta+root+test : type miss (requested=0x16, cached=0x13) 2023-01-24T17:05:13.159 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.152+0000 7f51d5404e40 10 cache put: name=default.rgw.meta+root+test info.flags=0x16 2023-01-24T17:05:13.159 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.152+0000 7f51d5404e40 10 moving default.rgw.meta+root+test to cache LRU end 2023-01-24T17:05:13.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.152+0000 7f51d5404e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:05:13.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.152+0000 7f51d5404e40 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:05:13.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.152+0000 7f51d5404e40 10 cache get: name=default.rgw.meta+root+test : hit (requested=0x11, cached=0x17) 2023-01-24T17:05:13.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.152+0000 7f51d5404e40 10 removing default.rgw.meta+root+test from cache 2023-01-24T17:05:13.160 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.152+0000 7f51d5404e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:test, ofs0, ns] 2023-01-24T17:05:13.161 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.152+0000 7f51d0f3b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562549 cookie 94372320285056 notifier 4247 bl.length()=148 2023-01-24T17:05:13.162 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.156+0000 7f51d5404e40 10 removing default.rgw.meta+root+.bucket.meta.test:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518 from cache 2023-01-24T17:05:13.163 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.156+0000 7f51d5404e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518, ofs0, ns] 2023-01-24T17:05:13.163 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.156+0000 7f51cbfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562550 cookie 94372320285056 notifier 4247 bl.length()=207 2023-01-24T17:05:13.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.176+0000 7f51d5404e40 10 removing default.rgw.log++pubsub..bucket.test/2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518 from cache 2023-01-24T17:05:13.185 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.176+0000 7f51d5404e40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test/2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.518, ofs0, ns] 2023-01-24T17:05:13.186 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.176+0000 7f51d0f3b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562347 cookie 94372320280784 notifier 4247 bl.length()=204 2023-01-24T17:05:13.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 20 get_system_obj_state: rctx=0x7ffe2833acf0 obj=default.rgw.meta:root:.bucket.meta.testbuck2:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519 state=0x55d4c489fef0 s->prefetch_data=0 2023-01-24T17:05:13.190 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519 : miss 2023-01-24T17:05:13.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519 info.flags=0x16 2023-01-24T17:05:13.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 10 adding default.rgw.meta+root+.bucket.meta.testbuck2:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519 to cache LRU end 2023-01-24T17:05:13.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:05:13.191 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:05:13.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:05:13.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 20 Read xattr: user.rgw.acl 2023-01-24T17:05:13.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 20 Read xattr: user.rgw.idtag 2023-01-24T17:05:13.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 10 cache get: name=default.rgw.meta+root+.bucket.meta.testbuck2:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:05:13.192 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 20 rados_obj.operate() r=0 bl.length=369 2023-01-24T17:05:13.193 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 10 cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519 info.flags=0x13 2023-01-24T17:05:13.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 10 moving default.rgw.meta+root+.bucket.meta.testbuck2:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519 to cache LRU end 2023-01-24T17:05:13.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:05:13.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 10 updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:05:13.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.testbuck2:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519 2023-01-24T17:05:13.194 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.184+0000 7f51d5404e40 10 int RGWRados::cls_bucket_list_unordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, uint32_t, bool, std::vector&, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter) :testbuck2[2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519]) start_after="", prefix="", shard_id=-1", num_entries=1100, list_versions=1, force_check_filter is unset 2023-01-24T17:05:13.197 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.188+0000 7f51d5404e40 10 cls_bucket_list_unordered: got 47EA07 2023-01-24T17:05:13.198 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.192+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=testbuck2:47EA07 state=0x55d4c4a95e58 s->prefetch_data=0 2023-01-24T17:05:13.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.192+0000 7f51d5404e40 10 manifest: total_size = 399155 2023-01-24T17:05:13.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.192+0000 7f51d5404e40 20 get_obj_state: setting s->obj_tag to 2ee3967b-537d-4521-a85e-a66f0a1f41f2.4205.15358143298697192595 2023-01-24T17:05:13.199 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.192+0000 7f51d5404e40 20 get_obj_state: rctx=0x7ffe2833ad10 obj=testbuck2:47EA07 state=0x55d4c4a95e58 s->prefetch_data=0 2023-01-24T17:05:13.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.192+0000 7f51d5404e40 20 prepare_atomic_modification: state is not atomic. state=0x55d4c4a95e58 2023-01-24T17:05:13.200 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.192+0000 7f51d5404e40 20 bucket index object: default.rgw.buckets.index:.dir.2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519.9 2023-01-24T17:05:13.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.196+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-source-hints.testbuck2 : miss 2023-01-24T17:05:13.203 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.196+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.196+0000 7f51d5404e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.196+0000 7f51d5404e40 10 cache put: name=default.rgw.log++bucket.sync-source-hints.testbuck2 info.flags=0x0 2023-01-24T17:05:13.204 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.196+0000 7f51d5404e40 10 adding default.rgw.log++bucket.sync-source-hints.testbuck2 to cache LRU end 2023-01-24T17:05:13.205 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.196+0000 7f51d5404e40 10 cache get: name=default.rgw.log++bucket.sync-target-hints.testbuck2 : miss 2023-01-24T17:05:13.205 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.196+0000 7f51d5404e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.205 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.196+0000 7f51d5404e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.206 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.196+0000 7f51d5404e40 10 cache put: name=default.rgw.log++bucket.sync-target-hints.testbuck2 info.flags=0x0 2023-01-24T17:05:13.206 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.196+0000 7f51d5404e40 10 adding default.rgw.log++bucket.sync-target-hints.testbuck2 to cache LRU end 2023-01-24T17:05:13.206 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.196+0000 7f51d5404e40 10 chain_cache_entry: cache_locator= 2023-01-24T17:05:13.206 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.196+0000 7f51d5404e40 20 chain_cache_entry: couldn't find cache locator 2023-01-24T17:05:13.206 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.196+0000 7f51d5404e40 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-01-24T17:05:13.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.196+0000 7f51d5404e40 20 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): starting attempt 1 2023-01-24T17:05:13.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.196+0000 7f51d5404e40 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): :testbuck2[2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519]) start_after="_multipart_", prefix="_multipart_, delimiter="", shard_id=-1", num_entries=1001, list_versions=0, expansion_factor=1, force_check_filter is unset 2023-01-24T17:05:13.207 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.196+0000 7f51d5404e40 10 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-01-24T17:05:13.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, count=0, is_truncated=0 2023-01-24T17:05:13.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 20 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): returning, last_entry NOT SET 2023-01-24T17:05:13.208 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): end of outer loop, truncated=0, count=0, attempt=1 2023-01-24T17:05:13.209 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 10 int RGWRados::Bucket::List::list_objects_ordered(const DoutPrefixProvider*, int64_t, std::vector*, std::map, bool>*, bool*, optional_yield): exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-01-24T17:05:13.209 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":testbuck2[2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519])"; uploads.size()=0; is_truncated=0 2023-01-24T17:05:13.209 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 10 lifecycle: RGWRados::convert_old_bucket_info(): bucket=:testbuck2[2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519]) 2023-01-24T17:05:13.209 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 20 lifecycle: get_system_obj_state: rctx=0x7ffe2833a7a8 obj=default.rgw.meta:root:testbuck2 state=0x55d4c489fef0 s->prefetch_data=0 2023-01-24T17:05:13.209 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 10 lifecycle: cache get: name=default.rgw.meta+root+testbuck2 : miss 2023-01-24T17:05:13.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 10 lifecycle: cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x16 2023-01-24T17:05:13.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 10 lifecycle: adding default.rgw.meta+root+testbuck2 to cache LRU end 2023-01-24T17:05:13.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:05:13.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 20 lifecycle: get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:05:13.210 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 20 lifecycle: Read xattr: user.rgw.idtag 2023-01-24T17:05:13.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 10 lifecycle: cache get: name=default.rgw.meta+root+testbuck2 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:05:13.211 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 20 lifecycle: rados->read ofs=0 len=0 2023-01-24T17:05:13.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 20 lifecycle: rados_obj.operate() r=0 bl.length=191 2023-01-24T17:05:13.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 10 lifecycle: cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x13 2023-01-24T17:05:13.212 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 10 lifecycle: moving default.rgw.meta+root+testbuck2 to cache LRU end 2023-01-24T17:05:13.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.200+0000 7f51d5404e40 10 lifecycle: updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:05:13.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.204+0000 7f51d5404e40 10 lifecycle: cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519 info.flags=0x17 2023-01-24T17:05:13.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.204+0000 7f51d5404e40 10 lifecycle: moving default.rgw.meta+root+.bucket.meta.testbuck2:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519 to cache LRU end 2023-01-24T17:05:13.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.204+0000 7f51d5404e40 10 lifecycle: updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:05:13.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.204+0000 7f51d5404e40 10 lifecycle: distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.testbuck2:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519, ofs0, ns] 2023-01-24T17:05:13.213 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.204+0000 7f51cbfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562551 cookie 94372320285056 notifier 4247 bl.length()=810 2023-01-24T17:05:13.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.204+0000 7f51cbfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.testbuck2:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519 info.flags=0x17 2023-01-24T17:05:13.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.204+0000 7f51cbfff700 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.testbuck2:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519 to cache LRU end 2023-01-24T17:05:13.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.204+0000 7f51cbfff700 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=185 2023-01-24T17:05:13.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.204+0000 7f51d5404e40 20 lifecycle: handle_bi_update(): bucket=:testbuck2[2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519]): orig_sources= new_sources= 2023-01-24T17:05:13.214 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.204+0000 7f51d5404e40 20 lifecycle: handle_bi_update(): bucket=:testbuck2[2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519]): potential sources added=[] removed=[] 2023-01-24T17:05:13.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.204+0000 7f51d5404e40 20 lifecycle: handle_bi_update(): bucket=:testbuck2[2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519]): orig_dests= new_dests= 2023-01-24T17:05:13.215 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.204+0000 7f51d5404e40 20 lifecycle: handle_bi_update(): bucket=:testbuck2[2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519]): potential dests added=[] removed=[] 2023-01-24T17:05:13.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.216+0000 7f51d5404e40 20 get_system_obj_state: rctx=0x7ffe2833a228 obj=default.rgw.meta:root:testbuck2 state=0x55d4c489fef0 s->prefetch_data=0 2023-01-24T17:05:13.223 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.216+0000 7f51d5404e40 10 cache get: name=default.rgw.meta+root+testbuck2 : type miss (requested=0x16, cached=0x13) 2023-01-24T17:05:13.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.216+0000 7f51d5404e40 10 cache put: name=default.rgw.meta+root+testbuck2 info.flags=0x16 2023-01-24T17:05:13.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.216+0000 7f51d5404e40 10 moving default.rgw.meta+root+testbuck2 to cache LRU end 2023-01-24T17:05:13.224 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.216+0000 7f51d5404e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:05:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.216+0000 7f51d5404e40 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:05:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.216+0000 7f51d5404e40 10 cache get: name=default.rgw.meta+root+testbuck2 : hit (requested=0x11, cached=0x17) 2023-01-24T17:05:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.216+0000 7f51d5404e40 10 removing default.rgw.meta+root+testbuck2 from cache 2023-01-24T17:05:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.216+0000 7f51d5404e40 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:testbuck2, ofs0, ns] 2023-01-24T17:05:13.225 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.216+0000 7f51cbfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562552 cookie 94372320282784 notifier 4247 bl.length()=153 2023-01-24T17:05:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.220+0000 7f51d5404e40 10 removing default.rgw.meta+root+.bucket.meta.testbuck2:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519 from cache 2023-01-24T17:05:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.220+0000 7f51d5404e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.testbuck2:2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519, ofs0, ns] 2023-01-24T17:05:13.227 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.220+0000 7f51d0f3b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562553 cookie 94372320285056 notifier 4247 bl.length()=212 2023-01-24T17:05:13.249 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.240+0000 7f51d5404e40 10 removing default.rgw.log++pubsub..bucket.testbuck2/2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519 from cache 2023-01-24T17:05:13.250 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.240+0000 7f51d5404e40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.testbuck2/2ee3967b-537d-4521-a85e-a66f0a1f41f2.4207.519, ofs0, ns] 2023-01-24T17:05:13.250 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.240+0000 7f51d0f3b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562348 cookie 94372320280784 notifier 4247 bl.length()=209 2023-01-24T17:05:13.254 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.248+0000 7f51d5404e40 10 removing key index: GMDOQETASXBLZQIFOCKN 2023-01-24T17:05:13.254 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.248+0000 7f51d5404e40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.keys:GMDOQETASXBLZQIFOCKN, ofs0, ns] 2023-01-24T17:05:13.255 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.248+0000 7f51cbfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562349 cookie 94372320291424 notifier 4247 bl.length()=170 2023-01-24T17:05:13.256 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.248+0000 7f51d5404e40 10 removing email index: foo.client.0+test@test.test 2023-01-24T17:05:13.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.248+0000 7f51d5404e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.email:foo.client.0+test@test.test, ofs0, ns] 2023-01-24T17:05:13.257 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.248+0000 7f51d0f3b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562554 cookie 94372320285056 notifier 4247 bl.length()=178 2023-01-24T17:05:13.259 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.252+0000 7f51d5404e40 10 removing user buckets index 2023-01-24T17:05:13.259 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.252+0000 7f51d5404e40 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0.buckets, ofs0, ns] 2023-01-24T17:05:13.260 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.252+0000 7f51cbfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562555 cookie 94372320272416 notifier 4247 bl.length()=169 2023-01-24T17:05:13.263 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.256+0000 7f51d5404e40 10 removing user index: foo.client.0 2023-01-24T17:05:13.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.256+0000 7f51d5404e40 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2023-01-24T17:05:13.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.256+0000 7f51d5404e40 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:users.uid:foo.client.0, ofs0, ns] 2023-01-24T17:05:13.264 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.256+0000 7f51d0f3b700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562350 cookie 94372320261536 notifier 4247 bl.length()=161 2023-01-24T17:05:13.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.260+0000 7f51d5404e40 20 remove_watcher() i=0 2023-01-24T17:05:13.268 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.260+0000 7f51d5404e40 2 removed watcher, disabling cache 2023-01-24T17:05:13.270 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.264+0000 7f51d5404e40 20 remove_watcher() i=1 2023-01-24T17:05:13.271 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.264+0000 7f51d5404e40 20 remove_watcher() i=2 2023-01-24T17:05:13.273 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.264+0000 7f51d5404e40 20 remove_watcher() i=3 2023-01-24T17:05:13.274 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.268+0000 7f51d5404e40 20 remove_watcher() i=4 2023-01-24T17:05:13.276 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.268+0000 7f51d5404e40 20 remove_watcher() i=5 2023-01-24T17:05:13.278 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.272+0000 7f51d5404e40 20 remove_watcher() i=6 2023-01-24T17:05:13.280 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.272+0000 7f51d5404e40 20 remove_watcher() i=7 2023-01-24T17:05:13.281 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.272+0000 7f51a3fe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:05:13.292 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2023-01-24T17:05:13.331 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:05:13.331 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:05:13.345 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.336+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.347 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.340+0000 7f44b6e2de40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.347 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.340+0000 7f44b6e2de40 20 realm 2023-01-24T17:05:13.347 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.340+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.340+0000 7f44b6e2de40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.348 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.340+0000 7f44b6e2de40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:05:13.349 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.340+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.349 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.340+0000 7f44b6e2de40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.349 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.340+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.350 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.340+0000 7f44b6e2de40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:05:13.350 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.340+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.350 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.344+0000 7f44b6e2de40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:05:13.351 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.344+0000 7f44b6e2de40 20 searching for the correct realm 2023-01-24T17:05:13.370 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.364+0000 7f44b6e2de40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:05:13.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.364+0000 7f44b6e2de40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:05:13.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.364+0000 7f44b6e2de40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:05:13.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.364+0000 7f44b6e2de40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:05:13.371 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.364+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.372 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.364+0000 7f44b6e2de40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.372 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.364+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.364+0000 7f44b6e2de40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:05:13.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.364+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.364+0000 7f44b6e2de40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:05:13.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.364+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.364+0000 7f44b6e2de40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.373 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.364+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.374 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.364+0000 7f44b6e2de40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.374 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.364+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.374 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.364+0000 7f44b6e2de40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:05:13.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:05:13.394 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:05:13.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:05:13.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:05:13.395 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:05:13.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:05:13.396 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:05:13.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.397 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:05:13.398 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 zone default found 2023-01-24T17:05:13.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 4 Realm: () 2023-01-24T17:05:13.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 4 ZoneGroup: default (0d4b5768-74c8-4719-bc1b-4550aa389781) 2023-01-24T17:05:13.399 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 4 Zone: default (2ee3967b-537d-4521-a85e-a66f0a1f41f2) 2023-01-24T17:05:13.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:05:13.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 zonegroup default 2023-01-24T17:05:13.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.388+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.392+0000 7f44b6e2de40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.400 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.392+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.401 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.392+0000 7f44b6e2de40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.401 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.392+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.401 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.392+0000 7f44b6e2de40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.402 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.392+0000 7f44b6e2de40 20 started sync module instance, tier type = 2023-01-24T17:05:13.402 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.392+0000 7f44b6e2de40 20 started zone id=2ee3967b-537d-4521-a85e-a66f0a1f41f2 (name=default) with tier type = 2023-01-24T17:05:13.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.412+0000 7f44b6e2de40 20 add_watcher() i=0 2023-01-24T17:05:13.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.412+0000 7f44b6e2de40 20 add_watcher() i=1 2023-01-24T17:05:13.418 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.412+0000 7f44b6e2de40 20 add_watcher() i=2 2023-01-24T17:05:13.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.412+0000 7f44b6e2de40 20 add_watcher() i=3 2023-01-24T17:05:13.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.412+0000 7f44b6e2de40 20 add_watcher() i=4 2023-01-24T17:05:13.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.412+0000 7f44b6e2de40 20 add_watcher() i=5 2023-01-24T17:05:13.419 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.412+0000 7f44b6e2de40 20 add_watcher() i=6 2023-01-24T17:05:13.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.412+0000 7f44b6e2de40 20 add_watcher() i=7 2023-01-24T17:05:13.420 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.412+0000 7f44b6e2de40 2 all 8 watchers are set, enabling cache 2023-01-24T17:05:13.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.416+0000 7f44857f2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:05:13.422 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.416+0000 7f44b6e2de40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:05:13.423 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.416+0000 7f44b6e2de40 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:05:13.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.512+0000 7f43f3fff700 20 reqs_thread_entry: start 2023-01-24T17:05:13.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.512+0000 7f43f1ffb700 20 reqs_thread_entry: start 2023-01-24T17:05:13.521 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.512+0000 7f44b6e2de40 20 init_complete bucket index max shards: 11 2023-01-24T17:05:13.522 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.512+0000 7f44b6e2de40 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:05:13.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.512+0000 7f43df7fe700 20 reqs_thread_entry: start 2023-01-24T17:05:13.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.512+0000 7f44b6e2de40 20 get_system_obj_state: rctx=0x7ffc34870f48 obj=default.rgw.meta:users.uid:bar.client.0 state=0x556fb91e4ef0 s->prefetch_data=0 2023-01-24T17:05:13.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.512+0000 7f44b6e2de40 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2023-01-24T17:05:13.523 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.512+0000 7f43dffff700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:05:43 2023 2023-01-24T17:05:13.523 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-01-24T17:05:13.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.516+0000 7f44b6e2de40 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2023-01-24T17:05:13.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.516+0000 7f44b6e2de40 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:05:13.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.516+0000 7f44b6e2de40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:05:13.524 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.516+0000 7f44b6e2de40 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:05:13.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.516+0000 7f44b6e2de40 20 Read xattr: user.rgw.idtag 2023-01-24T17:05:13.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.516+0000 7f44b6e2de40 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:05:13.525 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.516+0000 7f44b6e2de40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.526 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.516+0000 7f44b6e2de40 20 rados_obj.operate() r=0 bl.length=469 2023-01-24T17:05:13.526 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.516+0000 7f44b6e2de40 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x13 2023-01-24T17:05:13.526 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.516+0000 7f44b6e2de40 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2023-01-24T17:05:13.526 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.516+0000 7f44b6e2de40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:05:13.527 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.516+0000 7f44b6e2de40 10 removing key index: NJIHOWOUGIVKLZCJLGCP 2023-01-24T17:05:13.527 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.516+0000 7f44b6e2de40 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.keys:NJIHOWOUGIVKLZCJLGCP, ofs0, ns] 2023-01-24T17:05:13.527 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.516+0000 7f44b2964700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978561467 cookie 93938336971568 notifier 4253 bl.length()=170 2023-01-24T17:05:13.529 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.520+0000 7f44b6e2de40 10 removing email index: bar.client.0+test@test.test 2023-01-24T17:05:13.529 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.520+0000 7f44b6e2de40 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.email:bar.client.0+test@test.test, ofs0, ns] 2023-01-24T17:05:13.529 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.520+0000 7f44b2964700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562556 cookie 93938336970464 notifier 4253 bl.length()=178 2023-01-24T17:05:13.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.524+0000 7f44b6e2de40 10 removing user buckets index 2023-01-24T17:05:13.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.524+0000 7f44b6e2de40 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0.buckets, ofs0, ns] 2023-01-24T17:05:13.533 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.524+0000 7f44b2163700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978561468 cookie 93938336971568 notifier 4253 bl.length()=169 2023-01-24T17:05:13.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.528+0000 7f44b6e2de40 10 removing user index: bar.client.0 2023-01-24T17:05:13.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.528+0000 7f44b6e2de40 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2023-01-24T17:05:13.537 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.528+0000 7f44b6e2de40 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:users.uid:bar.client.0, ofs0, ns] 2023-01-24T17:05:13.538 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.528+0000 7f44b2964700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562557 cookie 93938336970464 notifier 4253 bl.length()=161 2023-01-24T17:05:13.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.536+0000 7f44b6e2de40 20 remove_watcher() i=0 2023-01-24T17:05:13.543 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.536+0000 7f44b6e2de40 2 removed watcher, disabling cache 2023-01-24T17:05:13.545 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.536+0000 7f44b6e2de40 20 remove_watcher() i=1 2023-01-24T17:05:13.547 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.540+0000 7f44b6e2de40 20 remove_watcher() i=2 2023-01-24T17:05:13.549 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.540+0000 7f44b6e2de40 20 remove_watcher() i=3 2023-01-24T17:05:13.552 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.544+0000 7f44b6e2de40 20 remove_watcher() i=4 2023-01-24T17:05:13.554 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.548+0000 7f44b6e2de40 20 remove_watcher() i=5 2023-01-24T17:05:13.556 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.548+0000 7f44b6e2de40 20 remove_watcher() i=6 2023-01-24T17:05:13.558 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.552+0000 7f44b6e2de40 20 remove_watcher() i=7 2023-01-24T17:05:13.560 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.552+0000 7f44857f2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:05:13.571 DEBUG:teuthology.orchestra.run.smithi103:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2023-01-24T17:05:13.610 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setuser ceph since I am not root 2023-01-24T17:05:13.610 INFO:teuthology.orchestra.run.smithi103.stderr:ignoring --setgroup ceph since I am not root 2023-01-24T17:05:13.622 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.612+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.624 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.616+0000 7f0183831e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.624 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.616+0000 7f0183831e40 20 realm 2023-01-24T17:05:13.625 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.616+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.616+0000 7f0183831e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.616+0000 7f0183831e40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-01-24T17:05:13.626 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.616+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.616+0000 7f0183831e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.616+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.616+0000 7f0183831e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:05:13.627 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.616+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.620+0000 7f0183831e40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:05:13.628 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.620+0000 7f0183831e40 20 searching for the correct realm 2023-01-24T17:05:13.649 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.640+0000 7f0183831e40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:05:13.649 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.640+0000 7f0183831e40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:05:13.649 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.640+0000 7f0183831e40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:05:13.649 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.640+0000 7f0183831e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:05:13.649 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.640+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.650 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.640+0000 7f0183831e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.651 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.640+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.651 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.640+0000 7f0183831e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:05:13.651 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.644+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.651 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.644+0000 7f0183831e40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:05:13.651 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.644+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.652 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.644+0000 7f0183831e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.652 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.644+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.652 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.644+0000 7f0183831e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.653 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.644+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.653 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.644+0000 7f0183831e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:05:13.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.668+0000 7f0183831e40 20 RGWRados::pool_iterate: got zonegroup_info.0d4b5768-74c8-4719-bc1b-4550aa389781 2023-01-24T17:05:13.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.668+0000 7f0183831e40 20 RGWRados::pool_iterate: got zone_info.2ee3967b-537d-4521-a85e-a66f0a1f41f2 2023-01-24T17:05:13.678 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.668+0000 7f0183831e40 20 RGWRados::pool_iterate: got zone_names.default 2023-01-24T17:05:13.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.668+0000 7f0183831e40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-01-24T17:05:13.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.668+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.679 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.668+0000 7f0183831e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:05:13.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.668+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.668+0000 7f0183831e40 20 rados_obj.operate() r=0 bl.length=873 2023-01-24T17:05:13.680 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.668+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.681 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 20 rados_obj.operate() r=0 bl.length=46 2023-01-24T17:05:13.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 20 rados_obj.operate() r=0 bl.length=362 2023-01-24T17:05:13.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.682 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 20 zone default found 2023-01-24T17:05:13.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 4 Realm: () 2023-01-24T17:05:13.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 4 ZoneGroup: default (0d4b5768-74c8-4719-bc1b-4550aa389781) 2023-01-24T17:05:13.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 4 Zone: default (2ee3967b-537d-4521-a85e-a66f0a1f41f2) 2023-01-24T17:05:13.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 10 cannot find current period zonegroup using local zonegroup configuration 2023-01-24T17:05:13.683 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 20 zonegroup default 2023-01-24T17:05:13.684 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.684 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.684 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.684 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.684 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.685 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 20 rados_obj.operate() r=-2 bl.length=0 2023-01-24T17:05:13.685 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 20 started sync module instance, tier type = 2023-01-24T17:05:13.685 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.672+0000 7f0183831e40 20 started zone id=2ee3967b-537d-4521-a85e-a66f0a1f41f2 (name=default) with tier type = 2023-01-24T17:05:13.705 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.696+0000 7f0183831e40 20 add_watcher() i=0 2023-01-24T17:05:13.706 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.696+0000 7f0183831e40 20 add_watcher() i=1 2023-01-24T17:05:13.706 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.696+0000 7f0183831e40 20 add_watcher() i=2 2023-01-24T17:05:13.706 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.696+0000 7f0183831e40 20 add_watcher() i=3 2023-01-24T17:05:13.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.696+0000 7f0183831e40 20 add_watcher() i=4 2023-01-24T17:05:13.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.696+0000 7f0183831e40 20 add_watcher() i=5 2023-01-24T17:05:13.707 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.696+0000 7f0183831e40 20 add_watcher() i=6 2023-01-24T17:05:13.708 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.700+0000 7f0183831e40 20 add_watcher() i=7 2023-01-24T17:05:13.709 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.700+0000 7f0183831e40 2 all 8 watchers are set, enabling cache 2023-01-24T17:05:13.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.704+0000 7f0183831e40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-01-24T17:05:13.710 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.704+0000 7f0183831e40 20 check_secure_mon_conn(): mode 1 is insecure 2023-01-24T17:05:13.711 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.704+0000 7f01527e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:05:13.796 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f00c17fa700 20 reqs_thread_entry: start 2023-01-24T17:05:13.796 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f00baffd700 20 reqs_thread_entry: start 2023-01-24T17:05:13.797 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f0183831e40 20 init_complete bucket index max shards: 11 2023-01-24T17:05:13.798 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f0183831e40 10 rgw notify: Started notification manager with: 1 workers 2023-01-24T17:05:13.798 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f00b97fa700 20 rgw notify: INFO: next queues processing will happen at: Tue Jan 24 17:05:44 2023 2023-01-24T17:05:13.798 INFO:teuthology.orchestra.run.smithi103.stderr: 2023-01-24T17:05:13.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f00b8ff9700 20 reqs_thread_entry: start 2023-01-24T17:05:13.799 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f0183831e40 20 get_system_obj_state: rctx=0x7ffe96bf5908 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x556d91490ef0 s->prefetch_data=0 2023-01-24T17:05:13.800 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f0183831e40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2023-01-24T17:05:13.800 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f0183831e40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2023-01-24T17:05:13.800 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f0183831e40 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:05:13.801 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f0183831e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:05:13.801 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f0183831e40 10 updating xattr: name=user.rgw.user-policy bl.length()=118 2023-01-24T17:05:13.801 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f0183831e40 20 get_system_obj_state: s->obj_tag was set empty 2023-01-24T17:05:13.802 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f0183831e40 20 Read xattr: user.rgw.idtag 2023-01-24T17:05:13.802 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f0183831e40 20 Read xattr: user.rgw.user-policy 2023-01-24T17:05:13.802 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f0183831e40 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x13, cached=0x16) 2023-01-24T17:05:13.803 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.788+0000 7f0183831e40 20 rados->read ofs=0 len=0 2023-01-24T17:05:13.804 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.792+0000 7f0183831e40 20 rados_obj.operate() r=0 bl.length=528 2023-01-24T17:05:13.804 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.792+0000 7f0183831e40 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x13 2023-01-24T17:05:13.804 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.792+0000 7f0183831e40 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2023-01-24T17:05:13.804 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.792+0000 7f0183831e40 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-01-24T17:05:13.805 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.792+0000 7f0183831e40 10 updating xattr: name=user.rgw.user-policy bl.length()=118 2023-01-24T17:05:13.805 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.792+0000 7f0183831e40 10 removing key index: FQIERVIEZQHTHXREXVKU 2023-01-24T17:05:13.805 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.792+0000 7f0183831e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.keys:FQIERVIEZQHTHXREXVKU, ofs0, ns] 2023-01-24T17:05:13.805 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.792+0000 7f0172ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562558 cookie 93929078762080 notifier 4259 bl.length()=170 2023-01-24T17:05:13.805 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.796+0000 7f0183831e40 10 removing email index: testx$tenanteduser.client.0+test@test.test 2023-01-24T17:05:13.806 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.796+0000 7f0183831e40 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test, ofs0, ns] 2023-01-24T17:05:13.806 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.796+0000 7f0172ffd700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562351 cookie 93929078763744 notifier 4259 bl.length()=193 2023-01-24T17:05:13.806 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.800+0000 7f0183831e40 10 removing user buckets index 2023-01-24T17:05:13.807 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.800+0000 7f0183831e40 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0.buckets, ofs0, ns] 2023-01-24T17:05:13.807 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.800+0000 7f01727fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562559 cookie 93929078762080 notifier 4259 bl.length()=184 2023-01-24T17:05:13.811 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.804+0000 7f0183831e40 10 removing user index: testx$tenanteduser.client.0 2023-01-24T17:05:13.811 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.804+0000 7f0183831e40 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2023-01-24T17:05:13.811 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.804+0000 7f0183831e40 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:testx$tenanteduser.client.0, ofs0, ns] 2023-01-24T17:05:13.812 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.804+0000 7f01727fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 188978562352 cookie 93929078760064 notifier 4259 bl.length()=176 2023-01-24T17:05:13.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.808+0000 7f0183831e40 20 remove_watcher() i=0 2023-01-24T17:05:13.817 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.808+0000 7f0183831e40 2 removed watcher, disabling cache 2023-01-24T17:05:13.819 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.812+0000 7f0183831e40 20 remove_watcher() i=1 2023-01-24T17:05:13.822 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.816+0000 7f0183831e40 20 remove_watcher() i=2 2023-01-24T17:05:13.824 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.816+0000 7f0183831e40 20 remove_watcher() i=3 2023-01-24T17:05:13.827 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.820+0000 7f0183831e40 20 remove_watcher() i=4 2023-01-24T17:05:13.829 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.820+0000 7f0183831e40 20 remove_watcher() i=5 2023-01-24T17:05:13.831 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.824+0000 7f0183831e40 20 remove_watcher() i=6 2023-01-24T17:05:13.834 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.828+0000 7f0183831e40 20 remove_watcher() i=7 2023-01-24T17:05:13.835 INFO:teuthology.orchestra.run.smithi103.stderr:2023-01-24T17:05:13.828+0000 7f01527e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-01-24T17:05:13.846 INFO:tasks.s3tests:Removing s3-tests... 2023-01-24T17:05:13.847 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf /home/ubuntu/cephtest/s3-tests 2023-01-24T17:05:13.961 DEBUG:teuthology.run_tasks:Unwinding manager s3tests 2023-01-24T17:05:13.978 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-01-24T17:05:14.015 DEBUG:tasks.rgw.client.0:waiting for process to exit 2023-01-24T17:05:14.016 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:05:14.016 INFO:tasks.rgw.client.0.smithi103.stdout:2023-01-24T17:05:14.008+0000 7f183ffff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=80 -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --rgw_crypt_kmip_addr 172.21.15.103:5696 --foreground (PID: 30081) UID: 0 2023-01-24T17:05:14.017 INFO:tasks.rgw.client.0.smithi103.stdout:2023-01-24T17:05:14.008+0000 7f18507dee40 -1 shutting down 2023-01-24T17:05:19.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2023-01-24T17:05:20.118 INFO:tasks.rgw.client.0:Stopped 2023-01-24T17:05:20.119 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2023-01-24T17:05:20.126 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/vault-root-token 2023-01-24T17:05:20.185 DEBUG:teuthology.run_tasks:Unwinding manager pykmip 2023-01-24T17:05:20.195 INFO:tasks.pykmip:Stopping PyKMIP instance 2023-01-24T17:05:20.195 DEBUG:tasks.pykmip.client.0:waiting for process to exit 2023-01-24T17:05:20.196 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:05:22.650 INFO:tasks.pykmip.client.0.smithi103.stderr:/home/ubuntu/cephtest/pykmip/kmip/services/server/crypto/engine.py:54: CryptographyDeprecationWarning: Blowfish has been deprecated 2023-01-24T17:05:22.650 INFO:tasks.pykmip.client.0.smithi103.stderr: enums.CryptographicAlgorithm.BLOWFISH: algorithms.Blowfish, 2023-01-24T17:05:22.651 INFO:tasks.pykmip.client.0.smithi103.stderr:/home/ubuntu/cephtest/pykmip/kmip/services/server/crypto/engine.py:56: CryptographyDeprecationWarning: CAST5 has been deprecated 2023-01-24T17:05:22.651 INFO:tasks.pykmip.client.0.smithi103.stderr: enums.CryptographicAlgorithm.CAST5: algorithms.CAST5, 2023-01-24T17:05:22.651 INFO:tasks.pykmip.client.0.smithi103.stderr:/home/ubuntu/cephtest/pykmip/kmip/services/server/crypto/engine.py:57: CryptographyDeprecationWarning: IDEA has been deprecated 2023-01-24T17:05:22.651 INFO:tasks.pykmip.client.0.smithi103.stderr: enums.CryptographicAlgorithm.IDEA: algorithms.IDEA, 2023-01-24T17:05:22.651 INFO:tasks.pykmip.client.0.smithi103.stderr:/home/ubuntu/cephtest/pykmip/kmip/services/server/engine.py:437: SAWarning: TypeDecorator EnumType() will not produce a cache key because the ``cache_ok`` attribute is not set to True. This can have significant performance implications including some performance degradations in comparison to prior SQLAlchemy versions. Set this attribute to True if this type object's state is safe to use in a cache key, or False to disable this warning. (Background on this error at: https://sqlalche.me/e/14/cprf) 2023-01-24T17:05:22.652 INFO:tasks.pykmip.client.0.smithi103.stderr: object_type = self._data_session.query( 2023-01-24T17:05:22.743 INFO:tasks.pykmip.client.0:Stopped 2023-01-24T17:05:22.743 INFO:tasks.pykmip:Removing system dependencies of PyKMIP... 2023-01-24T17:05:22.744 DEBUG:teuthology.orchestra.run.smithi103:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y purge libffi-dev 2023-01-24T17:05:22.811 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2023-01-24T17:05:22.999 INFO:teuthology.orchestra.run.smithi103.stdout:Building dependency tree... 2023-01-24T17:05:22.999 INFO:teuthology.orchestra.run.smithi103.stdout:Reading state information... 2023-01-24T17:05:23.162 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages were automatically installed and are no longer required: 2023-01-24T17:05:23.162 INFO:teuthology.orchestra.run.smithi103.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-01-24T17:05:23.163 INFO:teuthology.orchestra.run.smithi103.stdout:Use 'sudo apt autoremove' to remove them. 2023-01-24T17:05:23.187 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages will be REMOVED: 2023-01-24T17:05:23.188 INFO:teuthology.orchestra.run.smithi103.stdout: libffi-dev* 2023-01-24T17:05:23.739 INFO:teuthology.orchestra.run.smithi103.stdout:0 upgraded, 0 newly installed, 1 to remove and 117 not upgraded. 2023-01-24T17:05:23.740 INFO:teuthology.orchestra.run.smithi103.stdout:After this operation, 312 kB disk space will be freed. 2023-01-24T17:05:24.232 INFO:teuthology.orchestra.run.smithi103.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 153045 files and directories currently installed.) 2023-01-24T17:05:24.236 INFO:teuthology.orchestra.run.smithi103.stdout:Removing libffi-dev:amd64 (3.3-4) ... 2023-01-24T17:05:24.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 has been restored 2023-01-24T17:05:25.634 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2023-01-24T17:05:25.911 INFO:teuthology.orchestra.run.smithi103.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-01-24T17:05:27.540 DEBUG:teuthology.orchestra.run.smithi103:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y purge libsqlite3-dev 2023-01-24T17:05:27.611 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2023-01-24T17:05:27.801 INFO:teuthology.orchestra.run.smithi103.stdout:Building dependency tree... 2023-01-24T17:05:27.801 INFO:teuthology.orchestra.run.smithi103.stdout:Reading state information... 2023-01-24T17:05:27.984 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages were automatically installed and are no longer required: 2023-01-24T17:05:27.985 INFO:teuthology.orchestra.run.smithi103.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-01-24T17:05:27.985 INFO:teuthology.orchestra.run.smithi103.stdout:Use 'sudo apt autoremove' to remove them. 2023-01-24T17:05:28.014 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages will be REMOVED: 2023-01-24T17:05:28.014 INFO:teuthology.orchestra.run.smithi103.stdout: libsqlite3-dev* 2023-01-24T17:05:28.389 INFO:teuthology.orchestra.run.smithi103.stdout:0 upgraded, 0 newly installed, 1 to remove and 117 not upgraded. 2023-01-24T17:05:28.390 INFO:teuthology.orchestra.run.smithi103.stdout:After this operation, 2,372 kB disk space will be freed. 2023-01-24T17:05:28.502 INFO:teuthology.orchestra.run.smithi103.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 153011 files and directories currently installed.) 2023-01-24T17:05:28.507 INFO:teuthology.orchestra.run.smithi103.stdout:Removing libsqlite3-dev:amd64 (3.31.1-4ubuntu0.5) ... 2023-01-24T17:05:29.312 DEBUG:teuthology.orchestra.run.smithi103:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y purge libssl-dev 2023-01-24T17:05:29.381 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2023-01-24T17:05:29.569 INFO:teuthology.orchestra.run.smithi103.stdout:Building dependency tree... 2023-01-24T17:05:29.570 INFO:teuthology.orchestra.run.smithi103.stdout:Reading state information... 2023-01-24T17:05:29.732 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages were automatically installed and are no longer required: 2023-01-24T17:05:29.733 INFO:teuthology.orchestra.run.smithi103.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-01-24T17:05:29.733 INFO:teuthology.orchestra.run.smithi103.stdout:Use 'sudo apt autoremove' to remove them. 2023-01-24T17:05:29.763 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages will be REMOVED: 2023-01-24T17:05:29.764 INFO:teuthology.orchestra.run.smithi103.stdout: libssl-dev* 2023-01-24T17:05:30.143 INFO:teuthology.orchestra.run.smithi103.stdout:0 upgraded, 0 newly installed, 1 to remove and 117 not upgraded. 2023-01-24T17:05:30.144 INFO:teuthology.orchestra.run.smithi103.stdout:After this operation, 8,009 kB disk space will be freed. 2023-01-24T17:05:30.275 INFO:teuthology.orchestra.run.smithi103.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 153002 files and directories currently installed.) 2023-01-24T17:05:30.279 INFO:teuthology.orchestra.run.smithi103.stdout:Removing libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2023-01-24T17:05:31.125 DEBUG:teuthology.orchestra.run.smithi103:> DEBIAN_FRONTEND=noninteractive sudo -E apt-get -y purge python-dev 2023-01-24T17:05:31.200 INFO:teuthology.orchestra.run.smithi103.stdout:Reading package lists... 2023-01-24T17:05:31.389 INFO:teuthology.orchestra.run.smithi103.stdout:Building dependency tree... 2023-01-24T17:05:31.389 INFO:teuthology.orchestra.run.smithi103.stdout:Reading state information... 2023-01-24T17:05:31.559 INFO:teuthology.orchestra.run.smithi103.stdout:The following packages were automatically installed and are no longer required: 2023-01-24T17:05:31.560 INFO:teuthology.orchestra.run.smithi103.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-01-24T17:05:31.560 INFO:teuthology.orchestra.run.smithi103.stdout:Use 'sudo apt autoremove' to remove them. 2023-01-24T17:05:31.605 INFO:teuthology.orchestra.run.smithi103.stdout:0 upgraded, 0 newly installed, 0 to remove and 117 not upgraded. 2023-01-24T17:05:31.607 INFO:tasks.pykmip:Removing pykmip... 2023-01-24T17:05:31.608 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf /home/ubuntu/cephtest/pykmip 2023-01-24T17:05:31.665 DEBUG:teuthology.run_tasks:Unwinding manager openssl_keys 2023-01-24T17:05:31.675 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /home/ubuntu/cephtest/ca/kmiproot.crt /home/ubuntu/cephtest/ca/kmiproot.key 2023-01-24T17:05:31.681 DEBUG:teuthology.orchestra.run.smithi103:> rmdir --ignore-fail-on-non-empty /home/ubuntu/cephtest/ca 2023-01-24T17:05:31.747 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /home/ubuntu/cephtest/ca/kmip-client.crt /home/ubuntu/cephtest/ca/kmip-client.key 2023-01-24T17:05:31.754 DEBUG:teuthology.orchestra.run.smithi103:> rmdir --ignore-fail-on-non-empty /home/ubuntu/cephtest/ca 2023-01-24T17:05:31.761 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /home/ubuntu/cephtest/ca/kmip-server.crt /home/ubuntu/cephtest/ca/kmip-server.key 2023-01-24T17:05:31.767 DEBUG:teuthology.orchestra.run.smithi103:> rmdir --ignore-fail-on-non-empty /home/ubuntu/cephtest/ca 2023-01-24T17:05:31.771 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-01-24T17:05:31.784 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-01-24T17:05:31.785 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-24T17:05:32.087 INFO:teuthology.orchestra.run.smithi103.stdout: 2023-01-24T17:05:32.088 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":44,"fsid":"bf3b1473-4375-4731-8219-58951a76fe3f","created":"2023-01-24T16:58:58.666618+0000","modified":"2023-01-24T17:02:24.736631+0000","last_up_change":"2023-01-24T16:59:06.881220+0000","last_in_change":"2023-01-24T16:58:59.466576+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-24T16:59:06.853115+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-24T16:59:13.672144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2023-01-24T16:59:56.632027+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2023-01-24T16:59:58.133379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2023-01-24T17:00:00.155429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2023-01-24T17:00:01.925812+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2023-01-24T17:00:03.877157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2023-01-24T17:00:06.020757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"default.rgw.otp","create_time":"2023-01-24T17:00:16.354331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"default.rgw.buckets.non-ec","create_time":"2023-01-24T17:00:56.383065+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"d7b517d7-6f64-4813-8924-2a5c63538701","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":25680},{"type":"v1","addr":"172.21.15.103:6801","nonce":25680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":25680},{"type":"v1","addr":"172.21.15.103:6803","nonce":25680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":25680},{"type":"v1","addr":"172.21.15.103:6810","nonce":25680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":25680},{"type":"v1","addr":"172.21.15.103:6806","nonce":25680}]},"public_addr":"172.21.15.103:6801/25680","cluster_addr":"172.21.15.103:6803/25680","heartbeat_back_addr":"172.21.15.103:6810/25680","heartbeat_front_addr":"172.21.15.103:6806/25680","state":["exists","up"]},{"osd":1,"uuid":"f14ded86-755d-468d-85eb-95f1f6fb298a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":25682},{"type":"v1","addr":"172.21.15.103:6817","nonce":25682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":25682},{"type":"v1","addr":"172.21.15.103:6819","nonce":25682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":25682},{"type":"v1","addr":"172.21.15.103:6823","nonce":25682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":25682},{"type":"v1","addr":"172.21.15.103:6821","nonce":25682}]},"public_addr":"172.21.15.103:6817/25682","cluster_addr":"172.21.15.103:6819/25682","heartbeat_back_addr":"172.21.15.103:6823/25682","heartbeat_front_addr":"172.21.15.103:6821/25682","state":["exists","up"]},{"osd":2,"uuid":"2e034e0e-0171-4de2-80aa-2f7f6ef8c2c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6805","nonce":25683},{"type":"v1","addr":"172.21.15.103:6807","nonce":25683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6809","nonce":25683},{"type":"v1","addr":"172.21.15.103:6811","nonce":25683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":25683},{"type":"v1","addr":"172.21.15.103:6815","nonce":25683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":25683},{"type":"v1","addr":"172.21.15.103:6813","nonce":25683}]},"public_addr":"172.21.15.103:6807/25683","cluster_addr":"172.21.15.103:6811/25683","heartbeat_back_addr":"172.21.15.103:6815/25683","heartbeat_front_addr":"172.21.15.103:6813/25683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:59:03.885125+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:59:04.187806+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-01-24T16:59:03.725013+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":0,"to":2}]},{"pgid":"3.f","mappings":[{"from":0,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":2}]},{"pgid":"4.e","mappings":[{"from":1,"to":0}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":0}]},{"pgid":"4.2b","mappings":[{"from":1,"to":2}]},{"pgid":"4.30","mappings":[{"from":1,"to":2}]},{"pgid":"4.31","mappings":[{"from":1,"to":2}]},{"pgid":"10.c","mappings":[{"from":2,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-24T17:05:32.101 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-01-24T17:05:32.405 INFO:teuthology.misc:Shutting down mds daemons... 2023-01-24T17:05:32.406 INFO:teuthology.misc:Shutting down osd daemons... 2023-01-24T17:05:32.406 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-01-24T17:05:32.407 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:05:32.482 INFO:tasks.ceph.osd.0:Stopped 2023-01-24T17:05:32.482 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-01-24T17:05:32.483 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:05:32.558 INFO:tasks.ceph.osd.1:Stopped 2023-01-24T17:05:32.558 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-01-24T17:05:32.559 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:05:32.647 INFO:tasks.ceph.osd.2:Stopped 2023-01-24T17:05:32.647 INFO:teuthology.misc:Shutting down mgr daemons... 2023-01-24T17:05:32.648 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-01-24T17:05:32.648 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:05:32.696 INFO:tasks.ceph.mgr.x:Stopped 2023-01-24T17:05:32.696 INFO:teuthology.misc:Shutting down mon daemons... 2023-01-24T17:05:32.697 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-01-24T17:05:32.697 INFO:teuthology.orchestra.run:waiting for 300 2023-01-24T17:05:32.715 INFO:tasks.ceph.mon.a:Stopped 2023-01-24T17:05:32.716 INFO:tasks.ceph:Checking cluster log for badness... 2023-01-24T17:05:32.716 DEBUG:teuthology.orchestra.run.smithi103:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-01-24T17:05:32.729 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi103.front.sepia.ceph.com 2023-01-24T17:05:32.729 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-01-24T17:05:33.465 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi103.front.sepia.ceph.com 2023-01-24T17:05:33.465 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-01-24T17:05:33.771 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi103.front.sepia.ceph.com 2023-01-24T17:05:33.772 DEBUG:teuthology.orchestra.run.smithi103:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-01-24T17:05:33.913 INFO:tasks.ceph:Archiving mon data... 2023-01-24T17:05:33.914 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135469/data/mon.a.tgz 2023-01-24T17:05:33.915 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2023-01-24T17:05:33.920 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.o4CTDgXnb5 2023-01-24T17:05:33.920 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.o4CTDgXnb5 2023-01-24T17:05:34.057 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.o4CTDgXnb5 2023-01-24T17:05:34.082 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.o4CTDgXnb5 is 337KB 2023-01-24T17:05:34.140 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.o4CTDgXnb5 2023-01-24T17:05:34.147 INFO:tasks.ceph:Cleaning ceph cluster... 2023-01-24T17:05:34.147 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-01-24T17:05:34.163 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-01-24T17:05:34.163 DEBUG:teuthology.orchestra.run.smithi103:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-01-24T17:05:34.237 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-01-24T17:05:34.244 INFO:tasks.ceph:Archiving crash dumps... 2023-01-24T17:05:34.245 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135469/remote/smithi103/crash 2023-01-24T17:05:34.246 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2023-01-24T17:05:34.266 INFO:tasks.ceph:Compressing logs... 2023-01-24T17:05:34.267 DEBUG:teuthology.orchestra.run.smithi103:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-01-24T17:06:00.485 INFO:tasks.ceph:Archiving logs... 2023-01-24T17:06:00.486 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135469/remote/smithi103/log 2023-01-24T17:06:00.488 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /var/log/ceph -- . 2023-01-24T17:06:06.145 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-01-24T17:06:06.156 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2023-01-24T17:06:06.156 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2023-01-24T17:06:06.229 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-01-24T17:06:06.239 INFO:teuthology.task.clock:Checking final clock skew... 2023-01-24T17:06:06.239 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-24T17:06:06.259 INFO:teuthology.orchestra.run.smithi103.stdout: remote refid st t when poll reach delay offset jitter 2023-01-24T17:06:06.260 INFO:teuthology.orchestra.run.smithi103.stdout:============================================================================== 2023-01-24T17:06:06.260 INFO:teuthology.orchestra.run.smithi103.stdout:+hv01.front.sepi 216.240.36.24 3 u 23 64 377 0.070 -3.115 2.317 2023-01-24T17:06:06.260 INFO:teuthology.orchestra.run.smithi103.stdout:*hv02.front.sepi 152.44.36.235 3 u 23 64 377 0.074 1.828 2.571 2023-01-24T17:06:06.260 INFO:teuthology.orchestra.run.smithi103.stdout:+hv03.front.sepi 206.82.28.3 3 u 21 64 377 0.081 5.341 2.975 2023-01-24T17:06:06.261 INFO:teuthology.orchestra.run.smithi103.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-24T17:06:06.262 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-01-24T17:06:06.308 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-01-24T17:06:06.309 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-01-24T17:06:06.364 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-01-24T17:06:06.409 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-01-24T17:06:06.463 INFO:teuthology.task.internal:Duration was 838.754241 seconds 2023-01-24T17:06:06.464 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-01-24T17:06:06.513 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-01-24T17:06:06.514 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-01-24T17:06:06.569 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-01-24T17:06:06.569 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2023-01-24T17:06:06.570 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-01-24T17:06:06.582 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-01-24T17:06:06.583 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-01-24T17:06:06.606 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-01-24T17:06:06.607 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-01-24T17:06:06.734 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-01-24T17:06:06.744 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-01-24T17:06:06.745 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-01-24T17:06:06.761 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-01-24T17:06:06.819 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-01-24T17:06:06.835 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2023-01-24T17:06:06.850 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2023-01-24T17:06:06.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-24T17:06:06.856 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-01-24T17:06:06.867 INFO:teuthology.task.internal:Transferring archived files... 2023-01-24T17:06:06.868 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-01-24_16:04:03-rgw-wip-yuri7-testing-2023-01-23-1532-quincy-distro-default-smithi/7135469/remote/smithi103 2023-01-24T17:06:06.868 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2023-01-24T17:06:06.934 INFO:teuthology.task.internal:Removing archive directory... 2023-01-24T17:06:06.934 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2023-01-24T17:06:06.940 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-01-24T17:06:06.951 INFO:teuthology.task.internal:Not uploading archives. 2023-01-24T17:06:06.951 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-01-24T17:06:06.985 INFO:teuthology.task.internal:Tidying up after the test... 2023-01-24T17:06:06.985 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-01-24T17:06:06.993 INFO:teuthology.orchestra.run.smithi103.stdout: 924115 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 24 17:06 /home/ubuntu/cephtest 2023-01-24T17:06:06.995 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-01-24T17:06:07.087 INFO:teuthology.run:Summary data: description: rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} duration: 838.7542409896851 flavor: default owner: scheduled_yuriw@teuthology success: true 2023-01-24T17:06:07.088 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-24T17:06:07.155 INFO:teuthology.run:pass