2023-09-22T11:48:35.499 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-22T11:48:35.499 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_cfsnyder@teuthology --name cfsnyder-2023-09-22_11:28:34-rgw-wip-cfsnyder-bucket-check-pacific-distro-default-smithi --description rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} --archive /home/teuthworker/archive/cfsnyder-2023-09-22_11:28:34-rgw-wip-cfsnyder-bucket-check-pacific-distro-default-smithi/7400929 --verbose -- /home/teuthworker/archive/cfsnyder-2023-09-22_11:28:34-rgw-wip-cfsnyder-bucket-check-pacific-distro-default-smithi/7400929/orig.config.yaml 2023-09-22T11:48:35.537 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-22T11:48:35.613 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/cfsnyder-2023-09-22_11:28:34-rgw-wip-cfsnyder-bucket-check-pacific-distro-default-smithi/7400929 branch: wip-cfsnyder-bucket-check-pacific description: rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} email: null first_in_suite: false job_id: '7400929' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: cfsnyder-2023-09-22_11:28:34-rgw-wip-cfsnyder-bucket-check-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-cfsnyder-bucket-check-pacific ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= rgw lc debug interval: 10 rgw max chunk size: 4194304 rgw obj stripe size: 6291456 setgroup: ceph setuser: ceph global: mon client directed command retry: 5 ms inject socket failures: 5000 osd_max_pg_log_entries: 10 osd_min_pg_log_entries: 10 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 lockdep: true osd: debug ms: 1 debug osd: 20 lockdep: true osd objectstore: filestore osd shutdown pgref assert: true osd sloppy crc: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(POOL_APP_NOT_ENABLED\) - \(OSD_SLOW_PING_TIME sha1: 31c126b7c1c0c06b66d9d19c5bfc51c46b503392 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: flavor: default sha1: 31c126b7c1c0c06b66d9d19c5bfc51c46b503392 rgw: compression type: random ec-data-pool: false frontend: beast storage classes: LUKEWARM, FROZEN selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-cfsnyder-bucket-check-pacific sha1: 31c126b7c1c0c06b66d9d19c5bfc51c46b503392 owner: scheduled_cfsnyder@teuthology priority: 101 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b sha1: 31c126b7c1c0c06b66d9d19c5bfc51c46b503392 sleep_before_teardown: 0 suite: rgw suite_branch: wip-cfsnyder-bucket-check-pacific suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_31c126b7c1c0c06b66d9d19c5bfc51c46b503392/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 31c126b7c1c0c06b66d9d19c5bfc51c46b503392 targets: smithi096.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPqB9cWhFW7jYHtdBsk4nvELU9+BPYn+lLgm5zcLstUd8oFjM39ksSWoLh+5MZWrJckhoXwLLGmdqHubeLuPskw= smithi105.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEcE1zfd77YNZFbzP/oD1iV7nuNlVs4xW+tYZ1nrgTp5ZTad3T5piHy5PJibq2fKG/8lPvzIpja/Ox4ioWQ4Iq8= tasks: - install: null - ceph: null - openssl_keys: null - rgw: client.0: null - workunit: clients: client.0: - rgw/run-bucket-check.sh - workunit: clients: client.0: - cls/test_cls_lock.sh - cls/test_cls_log.sh - cls/test_cls_refcount.sh - cls/test_cls_rgw.sh - cls/test_cls_cmpomap.sh - cls/test_cls_2pc_queue.sh - rgw/test_rgw_gc_log.sh - rgw/test_rgw_obj.sh - rgw/test_rgw_throttle.sh - rgw/test_librgw_file.sh - ragweed: client.0: default-branch: ceph-pacific rgw_server: client.0 stages: prepare,check - workunit: clients: client.0: - rgw/run-reshard.sh - s3tests-java: client.0: force-branch: ceph-pacific force-repo: https://github.com/ceph/java_s3tests.git - s3tests: client.0: force-branch: ceph-pacific rgw_server: client.0 - workunit: clients: client.0: - rgw/run-versioning.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-22_11:28:34 tube: smithi user: cfsnyder verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-22T11:48:35.613 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_31c126b7c1c0c06b66d9d19c5bfc51c46b503392/qa; will attempt to use it 2023-09-22T11:48:35.614 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_31c126b7c1c0c06b66d9d19c5bfc51c46b503392/qa/tasks 2023-09-22T11:48:35.614 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-22T11:48:35.616 INFO:teuthology.task.internal:Checking packages... 2023-09-22T11:48:35.639 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '31c126b7c1c0c06b66d9d19c5bfc51c46b503392' 2023-09-22T11:48:35.639 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-22T11:48:35.640 INFO:teuthology.packaging:ref: None 2023-09-22T11:48:35.640 INFO:teuthology.packaging:tag: None 2023-09-22T11:48:35.640 INFO:teuthology.packaging:branch: wip-cfsnyder-bucket-check-pacific 2023-09-22T11:48:35.640 INFO:teuthology.packaging:sha1: 31c126b7c1c0c06b66d9d19c5bfc51c46b503392 2023-09-22T11:48:35.640 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-cfsnyder-bucket-check-pacific 2023-09-22T11:48:35.859 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-5.g31c126b7 2023-09-22T11:48:35.860 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-22T11:48:35.866 INFO:teuthology.task.internal:no buildpackages task found 2023-09-22T11:48:35.866 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-22T11:48:35.891 INFO:teuthology.task.internal:Saving configuration 2023-09-22T11:48:35.905 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-22T11:48:35.951 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-22T11:48:35.980 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/cfsnyder-2023-09-22_11:28:34-rgw-wip-cfsnyder-bucket-check-pacific-distro-default-smithi/7400929', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-22 11:43:31.361713', 'locked_by': 'scheduled_cfsnyder@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPqB9cWhFW7jYHtdBsk4nvELU9+BPYn+lLgm5zcLstUd8oFjM39ksSWoLh+5MZWrJckhoXwLLGmdqHubeLuPskw='} 2023-09-22T11:48:36.001 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/cfsnyder-2023-09-22_11:28:34-rgw-wip-cfsnyder-bucket-check-pacific-distro-default-smithi/7400929', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-22 11:43:31.363480', 'locked_by': 'scheduled_cfsnyder@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEcE1zfd77YNZFbzP/oD1iV7nuNlVs4xW+tYZ1nrgTp5ZTad3T5piHy5PJibq2fKG/8lPvzIpja/Ox4ioWQ4Iq8='} 2023-09-22T11:48:36.001 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-22T11:48:36.006 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-09-22T11:48:36.006 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-09-22T11:48:36.006 INFO:teuthology.run_tasks:Running task console_log... 2023-09-22T11:48:36.073 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff58328eee0>, signals=[15]) 2023-09-22T11:48:36.074 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-22T11:48:36.079 INFO:teuthology.task.internal:Opening connections... 2023-09-22T11:48:36.079 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi096.front.sepia.ceph.com 2023-09-22T11:48:36.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T11:48:36.147 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2023-09-22T11:48:36.148 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T11:48:36.217 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-22T11:48:36.223 DEBUG:teuthology.orchestra.run.smithi096:> uname -m 2023-09-22T11:48:36.245 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2023-09-22T11:48:36.245 DEBUG:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2023-09-22T11:48:36.306 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Stream" 2023-09-22T11:48:36.307 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="8" 2023-09-22T11:48:36.307 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2023-09-22T11:48:36.307 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2023-09-22T11:48:36.307 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="8" 2023-09-22T11:48:36.307 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el8" 2023-09-22T11:48:36.307 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-22T11:48:36.307 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2023-09-22T11:48:36.307 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-22T11:48:36.307 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://centos.org/" 2023-09-22T11:48:36.307 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-22T11:48:36.308 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-22T11:48:36.308 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-22T11:48:36.308 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2023-09-22T11:48:36.330 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2023-09-22T11:48:36.353 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2023-09-22T11:48:36.353 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2023-09-22T11:48:36.414 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="CentOS Stream" 2023-09-22T11:48:36.414 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="8" 2023-09-22T11:48:36.415 INFO:teuthology.orchestra.run.smithi105.stdout:ID="centos" 2023-09-22T11:48:36.415 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="rhel fedora" 2023-09-22T11:48:36.415 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="8" 2023-09-22T11:48:36.415 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el8" 2023-09-22T11:48:36.415 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-22T11:48:36.415 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2023-09-22T11:48:36.415 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-22T11:48:36.415 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://centos.org/" 2023-09-22T11:48:36.415 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-22T11:48:36.415 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-22T11:48:36.415 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-22T11:48:36.416 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2023-09-22T11:48:36.439 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-22T11:48:36.448 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-22T11:48:36.454 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-22T11:48:36.454 DEBUG:teuthology.orchestra.run.smithi096:> test '!' -e /home/ubuntu/cephtest 2023-09-22T11:48:36.457 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2023-09-22T11:48:36.477 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-22T11:48:36.483 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-22T11:48:36.484 DEBUG:teuthology.orchestra.run.smithi096:> test -z $(ls -A /var/lib/ceph) 2023-09-22T11:48:36.518 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2023-09-22T11:48:36.548 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-22T11:48:36.598 INFO:teuthology.run_tasks:Running task kernel... 2023-09-22T11:48:36.613 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-22T11:48:36.613 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-22T11:48:36.613 DEBUG:teuthology.orchestra.run.smithi096:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-22T11:48:36.614 DEBUG:teuthology.orchestra.run.smithi105:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-22T11:48:36.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T11:48:36.634 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2023-09-22T11:48:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T11:48:36.635 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2023-09-22T11:48:36.694 INFO:teuthology.orchestra.run.smithi096.stdout:4.18.0-489.el8.x86_64 2023-09-22T11:48:36.694 INFO:teuthology.task.kernel:Running kernel on smithi096: 4.18.0-489.el8.x86_64 2023-09-22T11:48:36.695 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum install -y kernel 2023-09-22T11:48:36.697 INFO:teuthology.orchestra.run.smithi105.stdout:4.18.0-489.el8.x86_64 2023-09-22T11:48:36.697 INFO:teuthology.task.kernel:Running kernel on smithi105: 4.18.0-489.el8.x86_64 2023-09-22T11:48:36.698 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum install -y kernel 2023-09-22T12:07:40.976 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 8 - AppStream 30 kB/s | 33 MB 19:01 2023-09-22T12:07:53.157 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 8 - BaseOS 7.6 MB/s | 47 MB 00:06 2023-09-22T12:08:00.332 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 8 - Extras 40 kB/s | 18 kB 00:00 2023-09-22T12:08:00.865 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2023-09-22T12:08:01.175 INFO:teuthology.orchestra.run.smithi096.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-09-22T12:08:02.845 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2023-09-22T12:08:06.625 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 421 kB/s | 24 kB 00:00 2023-09-22T12:08:07.366 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 8 - AppStream 29 kB/s | 33 MB 19:28 2023-09-22T12:08:10.134 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-22T12:08:10.134 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-22T12:08:10.397 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T12:08:10.397 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:08:10.398 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2023-09-22T12:08:10.398 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:08:10.398 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T12:08:10.398 INFO:teuthology.orchestra.run.smithi096.stdout: kernel x86_64 4.18.0-513.el8 baseos 10 M 2023-09-22T12:08:10.398 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2023-09-22T12:08:10.398 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core x86_64 4.18.0-513.el8 baseos 43 M 2023-09-22T12:08:10.398 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules x86_64 4.18.0-513.el8 baseos 35 M 2023-09-22T12:08:10.398 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:08:10.398 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T12:08:10.398 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:08:10.399 INFO:teuthology.orchestra.run.smithi096.stdout:Install 3 Packages 2023-09-22T12:08:10.399 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:08:10.399 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 88 M 2023-09-22T12:08:10.399 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 96 M 2023-09-22T12:08:10.399 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T12:08:11.602 INFO:teuthology.orchestra.run.smithi096.stdout:(1/3): kernel-4.18.0-513.el8.x86_64.rpm 9.9 MB/s | 10 MB 00:01 2023-09-22T12:08:12.552 INFO:teuthology.orchestra.run.smithi096.stdout:(2/3): kernel-modules-4.18.0-513.el8.x86_64.rpm 18 MB/s | 35 MB 00:01 2023-09-22T12:08:13.168 INFO:teuthology.orchestra.run.smithi096.stdout:(3/3): kernel-core-4.18.0-513.el8.x86_64.rpm 16 MB/s | 43 MB 00:02 2023-09-22T12:08:13.168 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T12:08:13.168 INFO:teuthology.orchestra.run.smithi096.stdout:Total 32 MB/s | 88 MB 00:02 2023-09-22T12:08:13.843 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T12:08:14.274 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T12:08:14.274 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T12:08:15.488 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T12:08:15.518 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T12:08:15.526 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 8 - BaseOS 22 MB/s | 47 MB 00:02 2023-09-22T12:08:18.317 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T12:08:18.790 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-core-4.18.0-513.el8.x86_64 1/3 2023-09-22T12:08:22.434 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-4.18.0-513.el8.x86_64 1/3 2023-09-22T12:08:22.482 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 8 - Extras 77 kB/s | 18 kB 00:00 2023-09-22T12:08:22.831 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-4.18.0-513.el8.x86_64 2/3 2023-09-22T12:08:23.008 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2023-09-22T12:08:23.326 INFO:teuthology.orchestra.run.smithi105.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-09-22T12:08:24.244 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 21 MB/s | 16 MB 00:00 2023-09-22T12:08:26.208 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-4.18.0-513.el8.x86_64 2/3 2023-09-22T12:08:26.537 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-4.18.0-513.el8.x86_64 3/3 2023-09-22T12:08:28.035 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 538 kB/s | 24 kB 00:00 2023-09-22T12:08:31.601 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-22T12:08:31.601 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-22T12:08:31.864 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T12:08:31.864 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:08:31.865 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2023-09-22T12:08:31.865 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:08:31.865 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-09-22T12:08:31.865 INFO:teuthology.orchestra.run.smithi105.stdout: kernel x86_64 4.18.0-513.el8 baseos 10 M 2023-09-22T12:08:31.865 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2023-09-22T12:08:31.865 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-core x86_64 4.18.0-513.el8 baseos 43 M 2023-09-22T12:08:31.865 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules x86_64 4.18.0-513.el8 baseos 35 M 2023-09-22T12:08:31.865 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:08:31.865 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-09-22T12:08:31.865 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:08:31.865 INFO:teuthology.orchestra.run.smithi105.stdout:Install 3 Packages 2023-09-22T12:08:31.866 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:08:31.866 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 88 M 2023-09-22T12:08:31.866 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 96 M 2023-09-22T12:08:31.867 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-09-22T12:08:32.920 INFO:teuthology.orchestra.run.smithi105.stdout:(1/3): kernel-4.18.0-513.el8.x86_64.rpm 9.9 MB/s | 10 MB 00:01 2023-09-22T12:08:34.195 INFO:teuthology.orchestra.run.smithi105.stdout:(2/3): kernel-modules-4.18.0-513.el8.x86_64.rpm 15 MB/s | 35 MB 00:02 2023-09-22T12:08:34.612 INFO:teuthology.orchestra.run.smithi105.stdout:(3/3): kernel-core-4.18.0-513.el8.x86_64.rpm 16 MB/s | 43 MB 00:02 2023-09-22T12:08:34.612 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-09-22T12:08:34.613 INFO:teuthology.orchestra.run.smithi105.stdout:Total 32 MB/s | 88 MB 00:02 2023-09-22T12:08:35.279 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-09-22T12:08:35.704 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-09-22T12:08:35.704 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-09-22T12:08:36.918 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-09-22T12:08:36.948 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-09-22T12:08:39.724 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-09-22T12:08:40.296 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-core-4.18.0-513.el8.x86_64 1/3 2023-09-22T12:08:43.910 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-4.18.0-513.el8.x86_64 1/3 2023-09-22T12:08:44.287 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-modules-4.18.0-513.el8.x86_64 2/3 2023-09-22T12:08:47.671 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-4.18.0-513.el8.x86_64 2/3 2023-09-22T12:08:47.965 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-4.18.0-513.el8.x86_64 3/3 2023-09-22T12:08:48.650 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-4.18.0-513.el8.x86_64 3/3 2023-09-22T12:08:48.673 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-4.18.0-513.el8.x86_64 3/3 2023-09-22T12:08:49.014 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-4.18.0-513.el8.x86_64 3/3 2023-09-22T12:08:49.014 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-4.18.0-513.el8.x86_64 1/3 2023-09-22T12:08:49.015 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-core-4.18.0-513.el8.x86_64 2/3 2023-09-22T12:08:50.233 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-4.18.0-513.el8.x86_64 3/3 2023-09-22T12:08:50.233 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:08:50.233 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T12:08:50.233 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-4.18.0-513.el8.x86_64 kernel-core-4.18.0-513.el8.x86_64 2023-09-22T12:08:50.233 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-4.18.0-513.el8.x86_64 2023-09-22T12:08:50.233 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:08:50.233 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T12:08:50.403 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2023-09-22T12:08:50.620 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-4.18.0-513.el8.x86_64 2023-09-22T12:08:50.621 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.el8.x86_64 2023-09-22T12:08:50.621 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-513.el8.x86_64 2023-09-22T12:08:50.621 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-22T12:08:50.621 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-22T12:08:50.621 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-22T12:08:50.621 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi096.front.sepia.ceph.com, path=None, version=distro) 2023-09-22T12:08:50.622 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2023-09-22T12:08:50.829 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-4.18.0-513.el8.x86_64 2023-09-22T12:08:50.829 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.el8.x86_64 2023-09-22T12:08:50.829 DEBUG:teuthology.orchestra.run.smithi096:> sudo rpm -qi grub2-tools 2023-09-22T12:08:50.862 INFO:teuthology.orchestra.run.smithi096.stdout:Name : grub2-tools 2023-09-22T12:08:50.862 INFO:teuthology.orchestra.run.smithi096.stdout:Epoch : 1 2023-09-22T12:08:50.863 INFO:teuthology.orchestra.run.smithi096.stdout:Version : 2.02 2023-09-22T12:08:50.863 INFO:teuthology.orchestra.run.smithi096.stdout:Release : 129.el8 2023-09-22T12:08:50.863 INFO:teuthology.orchestra.run.smithi096.stdout:Architecture: x86_64 2023-09-22T12:08:50.863 INFO:teuthology.orchestra.run.smithi096.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-22T12:08:50.863 INFO:teuthology.orchestra.run.smithi096.stdout:Group : System Environment/Base 2023-09-22T12:08:50.863 INFO:teuthology.orchestra.run.smithi096.stdout:Size : 9265586 2023-09-22T12:08:50.863 INFO:teuthology.orchestra.run.smithi096.stdout:License : GPLv3+ 2023-09-22T12:08:50.863 INFO:teuthology.orchestra.run.smithi096.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-22T12:08:50.863 INFO:teuthology.orchestra.run.smithi096.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-22T12:08:50.864 INFO:teuthology.orchestra.run.smithi096.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-22T12:08:50.864 INFO:teuthology.orchestra.run.smithi096.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-22T12:08:50.864 INFO:teuthology.orchestra.run.smithi096.stdout:Relocations : (not relocatable) 2023-09-22T12:08:50.864 INFO:teuthology.orchestra.run.smithi096.stdout:Packager : CentOS BuildSystem 2023-09-22T12:08:50.864 INFO:teuthology.orchestra.run.smithi096.stdout:Vendor : CentOS 2023-09-22T12:08:50.864 INFO:teuthology.orchestra.run.smithi096.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-22T12:08:50.864 INFO:teuthology.orchestra.run.smithi096.stdout:Summary : Support tools for GRUB. 2023-09-22T12:08:50.864 INFO:teuthology.orchestra.run.smithi096.stdout:Description : 2023-09-22T12:08:50.864 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:08:50.864 INFO:teuthology.orchestra.run.smithi096.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-22T12:08:50.864 INFO:teuthology.orchestra.run.smithi096.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-22T12:08:50.864 INFO:teuthology.orchestra.run.smithi096.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-22T12:08:50.865 INFO:teuthology.orchestra.run.smithi096.stdout:hardware devices. 2023-09-22T12:08:50.865 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:08:50.865 INFO:teuthology.orchestra.run.smithi096.stdout:This subpackage provides tools for support of all platforms. 2023-09-22T12:08:50.866 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-22T12:08:50.866 INFO:teuthology.task.kernel:Updating grub on smithi096 to boot 4.18.0-513.el8.x86_64 2023-09-22T12:08:50.866 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-22T12:08:52.297 INFO:teuthology.orchestra.run.smithi096.stderr:Generating grub configuration file ... 2023-09-22T12:08:54.529 INFO:teuthology.orchestra.run.smithi096.stderr:done 2023-09-22T12:08:54.535 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2023-09-22T12:08:54.558 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.gYAgpccvyq 2023-09-22T12:08:54.558 DEBUG:teuthology.orchestra.run.smithi096:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gYAgpccvyq 2023-09-22T12:08:54.644 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.gYAgpccvyq 2023-09-22T12:08:54.769 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.gYAgpccvyq is 6KB 2023-09-22T12:08:54.792 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.gYAgpccvyq 2023-09-22T12:08:54.809 DEBUG:teuthology.orchestra.run.smithi096:> sudo /bin/ls /boot/loader/entries || true 2023-09-22T12:08:54.882 INFO:teuthology.orchestra.run.smithi096.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-22T12:08:54.882 INFO:teuthology.orchestra.run.smithi096.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-22T12:08:54.882 INFO:teuthology.orchestra.run.smithi096.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-22T12:08:54.882 INFO:teuthology.orchestra.run.smithi096.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-513.el8.x86_64.conf 2023-09-22T12:08:54.883 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-513.el8.x86_64 2023-09-22T12:08:55.042 DEBUG:teuthology.orchestra.run.smithi096:> sudo shutdown -r now 2023-09-22T12:09:09.877 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-4.18.0-513.el8.x86_64 3/3 2023-09-22T12:09:09.901 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-4.18.0-513.el8.x86_64 3/3 2023-09-22T12:09:10.228 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-4.18.0-513.el8.x86_64 3/3 2023-09-22T12:09:10.228 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-4.18.0-513.el8.x86_64 1/3 2023-09-22T12:09:10.228 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-core-4.18.0-513.el8.x86_64 2/3 2023-09-22T12:09:11.013 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-modules-4.18.0-513.el8.x86_64 3/3 2023-09-22T12:09:11.013 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:09:11.013 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-09-22T12:09:11.014 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-4.18.0-513.el8.x86_64 kernel-core-4.18.0-513.el8.x86_64 2023-09-22T12:09:11.014 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-4.18.0-513.el8.x86_64 2023-09-22T12:09:11.014 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:09:11.014 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T12:09:11.702 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2023-09-22T12:09:11.917 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-4.18.0-513.el8.x86_64 2023-09-22T12:09:11.917 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.el8.x86_64 2023-09-22T12:09:11.917 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-513.el8.x86_64 2023-09-22T12:09:11.918 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-22T12:09:11.918 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-22T12:09:11.918 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-09-22T12:09:11.918 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi105.front.sepia.ceph.com, path=None, version=distro) 2023-09-22T12:09:11.918 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2023-09-22T12:09:12.126 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-4.18.0-513.el8.x86_64 2023-09-22T12:09:12.126 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-513.el8.x86_64 2023-09-22T12:09:12.126 DEBUG:teuthology.orchestra.run.smithi105:> sudo rpm -qi grub2-tools 2023-09-22T12:09:12.159 INFO:teuthology.orchestra.run.smithi105.stdout:Name : grub2-tools 2023-09-22T12:09:12.159 INFO:teuthology.orchestra.run.smithi105.stdout:Epoch : 1 2023-09-22T12:09:12.159 INFO:teuthology.orchestra.run.smithi105.stdout:Version : 2.02 2023-09-22T12:09:12.159 INFO:teuthology.orchestra.run.smithi105.stdout:Release : 129.el8 2023-09-22T12:09:12.160 INFO:teuthology.orchestra.run.smithi105.stdout:Architecture: x86_64 2023-09-22T12:09:12.160 INFO:teuthology.orchestra.run.smithi105.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-22T12:09:12.160 INFO:teuthology.orchestra.run.smithi105.stdout:Group : System Environment/Base 2023-09-22T12:09:12.160 INFO:teuthology.orchestra.run.smithi105.stdout:Size : 9265586 2023-09-22T12:09:12.160 INFO:teuthology.orchestra.run.smithi105.stdout:License : GPLv3+ 2023-09-22T12:09:12.160 INFO:teuthology.orchestra.run.smithi105.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-22T12:09:12.160 INFO:teuthology.orchestra.run.smithi105.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-22T12:09:12.160 INFO:teuthology.orchestra.run.smithi105.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-22T12:09:12.160 INFO:teuthology.orchestra.run.smithi105.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-22T12:09:12.160 INFO:teuthology.orchestra.run.smithi105.stdout:Relocations : (not relocatable) 2023-09-22T12:09:12.160 INFO:teuthology.orchestra.run.smithi105.stdout:Packager : CentOS BuildSystem 2023-09-22T12:09:12.160 INFO:teuthology.orchestra.run.smithi105.stdout:Vendor : CentOS 2023-09-22T12:09:12.160 INFO:teuthology.orchestra.run.smithi105.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-22T12:09:12.161 INFO:teuthology.orchestra.run.smithi105.stdout:Summary : Support tools for GRUB. 2023-09-22T12:09:12.161 INFO:teuthology.orchestra.run.smithi105.stdout:Description : 2023-09-22T12:09:12.161 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:09:12.161 INFO:teuthology.orchestra.run.smithi105.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-22T12:09:12.161 INFO:teuthology.orchestra.run.smithi105.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-22T12:09:12.161 INFO:teuthology.orchestra.run.smithi105.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-22T12:09:12.161 INFO:teuthology.orchestra.run.smithi105.stdout:hardware devices. 2023-09-22T12:09:12.161 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:09:12.161 INFO:teuthology.orchestra.run.smithi105.stdout:This subpackage provides tools for support of all platforms. 2023-09-22T12:09:12.162 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-22T12:09:12.162 INFO:teuthology.task.kernel:Updating grub on smithi105 to boot 4.18.0-513.el8.x86_64 2023-09-22T12:09:12.162 DEBUG:teuthology.orchestra.run.smithi105:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-22T12:09:13.638 INFO:teuthology.orchestra.run.smithi105.stderr:Generating grub configuration file ... 2023-09-22T12:09:15.612 INFO:teuthology.orchestra.run.smithi105.stderr:done 2023-09-22T12:09:15.613 DEBUG:teuthology.orchestra.run.smithi105:> mktemp 2023-09-22T12:09:15.657 INFO:teuthology.orchestra.run.smithi105.stdout:/tmp/tmp.oC0Baht3I8 2023-09-22T12:09:15.658 DEBUG:teuthology.orchestra.run.smithi105:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.oC0Baht3I8 2023-09-22T12:09:15.731 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0666 /tmp/tmp.oC0Baht3I8 2023-09-22T12:09:15.958 DEBUG:teuthology.orchestra.remote:smithi105:/tmp/tmp.oC0Baht3I8 is 6KB 2023-09-22T12:09:16.008 DEBUG:teuthology.orchestra.run.smithi105:> rm -fr /tmp/tmp.oC0Baht3I8 2023-09-22T12:09:16.026 DEBUG:teuthology.orchestra.run.smithi105:> sudo /bin/ls /boot/loader/entries || true 2023-09-22T12:09:16.104 INFO:teuthology.orchestra.run.smithi105.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-22T12:09:16.104 INFO:teuthology.orchestra.run.smithi105.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-22T12:09:16.104 INFO:teuthology.orchestra.run.smithi105.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-22T12:09:16.104 INFO:teuthology.orchestra.run.smithi105.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-513.el8.x86_64.conf 2023-09-22T12:09:16.105 DEBUG:teuthology.orchestra.run.smithi105:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-513.el8.x86_64 2023-09-22T12:09:16.298 DEBUG:teuthology.orchestra.run.smithi105:> sudo shutdown -r now 2023-09-22T12:09:25.050 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-22T12:09:25.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2023-09-22T12:09:25.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T12:09:43.546 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2023-09-22T12:09:46.302 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-09-22T12:09:46.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2023-09-22T12:09:46.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T12:09:52.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2023-09-22T12:09:52.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T12:09:55.617 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2023-09-22T12:10:04.794 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2023-09-22T12:10:07.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2023-09-22T12:10:07.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T12:10:13.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2023-09-22T12:10:13.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T12:10:16.858 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2023-09-22T12:10:26.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2023-09-22T12:10:28.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2023-09-22T12:10:28.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T12:10:41.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2023-09-22T12:10:41.059 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T12:10:41.421 DEBUG:teuthology.orchestra.run.smithi096:> true 2023-09-22T12:10:41.665 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2023-09-22T12:10:41.665 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-513.el8.x86_64"... 2023-09-22T12:10:41.665 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2023-09-22T12:10:41.683 INFO:teuthology.orchestra.run.smithi096.stdout:4.18.0-513.el8.x86_64 2023-09-22T12:10:41.684 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-513.el8.x86_64 vs 4.18.0-513.el8.x86_64 2023-09-22T12:10:41.684 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-22T12:10:41.684 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-22T12:10:42.685 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-22T12:10:42.685 DEBUG:teuthology.orchestra.run.smithi096:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-22T12:10:42.939 INFO:teuthology.orchestra.run.smithi096.stdout:ttyS1 2023-09-22T12:10:42.961 DEBUG:teuthology.parallel:result is None 2023-09-22T12:11:00.732 DEBUG:teuthology.orchestra.run.smithi105:> true 2023-09-22T12:11:00.982 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2023-09-22T12:11:00.982 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-513.el8.x86_64"... 2023-09-22T12:11:00.983 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2023-09-22T12:11:01.039 INFO:teuthology.orchestra.run.smithi105.stdout:4.18.0-513.el8.x86_64 2023-09-22T12:11:01.039 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-513.el8.x86_64 vs 4.18.0-513.el8.x86_64 2023-09-22T12:11:01.039 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-22T12:11:01.039 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-22T12:11:02.041 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-09-22T12:11:02.041 DEBUG:teuthology.orchestra.run.smithi105:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-22T12:11:02.121 INFO:teuthology.orchestra.run.smithi105.stdout:ttyS1 2023-09-22T12:11:02.137 DEBUG:teuthology.parallel:result is None 2023-09-22T12:11:02.137 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-22T12:11:02.144 INFO:teuthology.task.internal:Creating test directory... 2023-09-22T12:11:02.145 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-22T12:11:02.148 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-22T12:11:02.169 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-22T12:11:02.176 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-22T12:11:02.283 INFO:teuthology.task.internal:Creating archive directory... 2023-09-22T12:11:02.284 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-22T12:11:02.286 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-22T12:11:02.331 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-22T12:11:02.342 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-22T12:11:02.342 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-22T12:11:02.368 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-22T12:11:02.406 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-22T12:11:02.411 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-22T12:11:02.418 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-22T12:11:02.422 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-22T12:11:02.423 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-22T12:11:02.430 INFO:teuthology.task.internal:Configuring sudo... 2023-09-22T12:11:02.430 DEBUG:teuthology.orchestra.run.smithi096:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-22T12:11:02.462 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-22T12:11:02.493 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-22T12:11:02.505 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-22T12:11:02.505 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-22T12:11:02.533 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-22T12:11:02.552 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-22T12:11:02.635 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-22T12:11:02.684 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-22T12:11:02.767 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-22T12:11:02.812 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T12:11:02.813 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-22T12:11:02.880 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-22T12:11:02.926 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-22T12:11:02.979 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-22T12:11:03.065 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-22T12:11:03.110 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T12:11:03.110 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-22T12:11:03.177 DEBUG:teuthology.orchestra.run.smithi096:> sudo service rsyslog restart 2023-09-22T12:11:03.180 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2023-09-22T12:11:03.245 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-22T12:11:03.297 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-22T12:11:03.694 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-22T12:11:03.701 INFO:teuthology.task.internal:Starting timer... 2023-09-22T12:11:03.701 INFO:teuthology.run_tasks:Running task pcp... 2023-09-22T12:11:03.735 INFO:teuthology.run_tasks:Running task selinux... 2023-09-22T12:11:03.759 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-22T12:11:03.760 DEBUG:teuthology.orchestra.run.smithi096:> sudo service auditd rotate 2023-09-22T12:11:03.845 INFO:teuthology.orchestra.run.smithi096.stdout:Rotating logs: 2023-09-22T12:11:03.846 DEBUG:teuthology.orchestra.run.smithi105:> sudo service auditd rotate 2023-09-22T12:11:03.919 INFO:teuthology.orchestra.run.smithi105.stdout:Rotating logs: 2023-09-22T12:11:03.921 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-22T12:11:03.921 DEBUG:teuthology.orchestra.run.smithi096:> /usr/sbin/getenforce 2023-09-22T12:11:03.955 INFO:teuthology.orchestra.run.smithi096.stdout:Permissive 2023-09-22T12:11:03.955 DEBUG:teuthology.orchestra.run.smithi105:> /usr/sbin/getenforce 2023-09-22T12:11:03.976 INFO:teuthology.orchestra.run.smithi105.stdout:Permissive 2023-09-22T12:11:03.976 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi096.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi105.front.sepia.ceph.com': 'permissive'} 2023-09-22T12:11:03.977 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-22T12:11:04.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T12:11:04.024 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-22T12:11:04.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T12:11:04.052 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-22T12:11:04.053 DEBUG:teuthology.orchestra.run.smithi096:> sudo /usr/sbin/setenforce permissive 2023-09-22T12:11:04.091 DEBUG:teuthology.orchestra.run.smithi105:> sudo /usr/sbin/setenforce permissive 2023-09-22T12:11:04.120 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-22T12:11:04.129 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-09-22T12:11:04.134 INFO:teuthology.repo_utils:Fetching main from origin 2023-09-22T12:11:04.276 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-22T12:11:04.292 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-22T12:11:04.293 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi096.front.sepia.ceph.com,smithi105.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-22T12:31:48.145 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi096.front.sepia.ceph.com'), Remote(name='ubuntu@smithi105.front.sepia.ceph.com')] 2023-09-22T12:31:48.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2023-09-22T12:31:48.148 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T12:31:48.215 DEBUG:teuthology.orchestra.run.smithi096:> true 2023-09-22T12:31:48.285 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2023-09-22T12:31:48.285 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2023-09-22T12:31:48.285 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-22T12:31:48.354 DEBUG:teuthology.orchestra.run.smithi105:> true 2023-09-22T12:31:48.420 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2023-09-22T12:31:48.421 INFO:teuthology.run_tasks:Running task clock... 2023-09-22T12:31:48.442 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-22T12:31:48.442 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-22T12:31:48.443 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-22T12:31:48.445 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-22T12:31:48.445 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-22T12:31:48.476 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-22T12:31:48.491 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-22T12:31:48.493 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-22T12:31:48.507 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-22T12:31:48.557 INFO:teuthology.orchestra.run.smithi096.stderr:sudo: ntpd: command not found 2023-09-22T12:31:48.569 INFO:teuthology.orchestra.run.smithi096.stdout:506 Cannot talk to daemon 2023-09-22T12:31:48.579 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: ntpd: command not found 2023-09-22T12:31:48.583 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-22T12:31:48.591 INFO:teuthology.orchestra.run.smithi105.stdout:506 Cannot talk to daemon 2023-09-22T12:31:48.596 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-22T12:31:48.605 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-22T12:31:48.618 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-22T12:31:48.669 INFO:teuthology.orchestra.run.smithi096.stderr:bash: ntpq: command not found 2023-09-22T12:31:48.672 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-22T12:31:48.672 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2023-09-22T12:31:48.672 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-22T12:31:48.673 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-22T12:31:48.673 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-22T12:31:48.673 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-22T12:31:48.694 INFO:teuthology.orchestra.run.smithi105.stderr:bash: ntpq: command not found 2023-09-22T12:31:48.698 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-22T12:31:48.699 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2023-09-22T12:31:48.699 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-22T12:31:48.699 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-22T12:31:48.699 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-22T12:31:48.699 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-22T12:31:48.699 INFO:teuthology.run_tasks:Running task install... 2023-09-22T12:31:48.708 DEBUG:teuthology.task.install:project ceph 2023-09-22T12:31:48.708 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '31c126b7c1c0c06b66d9d19c5bfc51c46b503392'}} 2023-09-22T12:31:48.708 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '31c126b7c1c0c06b66d9d19c5bfc51c46b503392'} 2023-09-22T12:31:48.708 INFO:teuthology.task.install:Using flavor: default 2023-09-22T12:31:48.714 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-22T12:31:48.714 INFO:teuthology.task.install:extra packages: [] 2023-09-22T12:31:48.714 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '31c126b7c1c0c06b66d9d19c5bfc51c46b503392', 'tag': None, 'wait_for_package': False} 2023-09-22T12:31:48.715 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=31c126b7c1c0c06b66d9d19c5bfc51c46b503392 2023-09-22T12:31:48.716 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '31c126b7c1c0c06b66d9d19c5bfc51c46b503392', 'tag': None, 'wait_for_package': False} 2023-09-22T12:31:48.717 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=31c126b7c1c0c06b66d9d19c5bfc51c46b503392 2023-09-22T12:31:48.867 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-cfsnyder-bucket-check-pacific/31c126b7c1c0c06b66d9d19c5bfc51c46b503392/centos/8/flavors/default/ 2023-09-22T12:31:48.867 INFO:teuthology.task.install.rpm:Package version is 16.2.14-5.g31c126b7 2023-09-22T12:31:49.068 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-cfsnyder-bucket-check-pacific/31c126b7c1c0c06b66d9d19c5bfc51c46b503392/centos/8/flavors/default/ 2023-09-22T12:31:49.068 INFO:teuthology.task.install.rpm:Package version is 16.2.14-5.g31c126b7 2023-09-22T12:31:49.156 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-cfsnyder-bucket-check-pacific/31c126b7c1c0c06b66d9d19c5bfc51c46b503392/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-cfsnyder-bucket-check-pacific/31c126b7c1c0c06b66d9d19c5bfc51c46b503392/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-cfsnyder-bucket-check-pacific/31c126b7c1c0c06b66d9d19c5bfc51c46b503392/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-22T12:31:49.156 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T12:31:49.157 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-22T12:31:49.174 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-cfsnyder-bucket-check-pacific/31c126b7c1c0c06b66d9d19c5bfc51c46b503392/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-cfsnyder-bucket-check-pacific/31c126b7c1c0c06b66d9d19c5bfc51c46b503392/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-cfsnyder-bucket-check-pacific/31c126b7c1c0c06b66d9d19c5bfc51c46b503392/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-22T12:31:49.174 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T12:31:49.174 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-22T12:31:49.186 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-22T12:31:49.186 DEBUG:teuthology.orchestra.run.smithi096:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/31c126b7c1c0c06b66d9d19c5bfc51c46b503392/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-22T12:31:49.205 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-22T12:31:49.205 DEBUG:teuthology.orchestra.run.smithi105:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/31c126b7c1c0c06b66d9d19c5bfc51c46b503392/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-22T12:31:49.254 DEBUG:teuthology.orchestra.run.smithi096:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-22T12:31:49.276 DEBUG:teuthology.orchestra.run.smithi105:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-22T12:31:49.332 DEBUG:teuthology.orchestra.run.smithi096:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-22T12:31:49.355 DEBUG:teuthology.orchestra.run.smithi105:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-22T12:31:49.398 INFO:teuthology.orchestra.run.smithi096.stdout:check_obsoletes = 1 2023-09-22T12:31:49.399 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum clean all 2023-09-22T12:31:49.423 INFO:teuthology.orchestra.run.smithi105.stdout:check_obsoletes = 1 2023-09-22T12:31:49.424 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum clean all 2023-09-22T12:31:49.677 INFO:teuthology.orchestra.run.smithi096.stdout:76 files removed 2023-09-22T12:31:49.704 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-radosgw 2023-09-22T12:31:49.715 INFO:teuthology.orchestra.run.smithi105.stdout:76 files removed 2023-09-22T12:31:49.742 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-radosgw 2023-09-22T12:31:50.770 INFO:teuthology.orchestra.run.smithi096.stdout:ceph packages for x86_64 115 kB/s | 76 kB 00:00 2023-09-22T12:31:50.800 INFO:teuthology.orchestra.run.smithi105.stdout:ceph packages for x86_64 117 kB/s | 76 kB 00:00 2023-09-22T12:31:51.188 INFO:teuthology.orchestra.run.smithi096.stdout:ceph noarch packages 40 kB/s | 15 kB 00:00 2023-09-22T12:31:51.193 INFO:teuthology.orchestra.run.smithi105.stdout:ceph noarch packages 43 kB/s | 15 kB 00:00 2023-09-22T12:31:51.589 INFO:teuthology.orchestra.run.smithi096.stdout:ceph source packages 3.9 kB/s | 1.4 kB 00:00 2023-09-22T12:31:51.718 INFO:teuthology.orchestra.run.smithi105.stdout:ceph source packages 2.9 kB/s | 1.4 kB 00:00 2023-09-22T12:49:47.807 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 8 - AppStream 32 kB/s | 33 MB 17:56 2023-09-22T12:49:57.963 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 47 MB 00:04 2023-09-22T12:50:08.455 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 8 - Extras 4.7 kB/s | 18 kB 00:03 2023-09-22T12:50:09.096 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 8 - Extras common packages 15 kB/s | 6.9 kB 00:00 2023-09-22T12:50:10.815 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 8 - PowerTools 4.1 MB/s | 6.5 MB 00:01 2023-09-22T12:50:12.241 INFO:teuthology.orchestra.run.smithi096.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-09-22T12:50:13.703 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-09-22T12:50:17.491 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 479 kB/s | 24 kB 00:00 2023-09-22T12:50:21.905 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T12:50:21.908 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:50:21.908 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2023-09-22T12:50:21.908 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:50:21.908 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T12:50:21.908 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw x86_64 2:16.2.14-5.g31c126b7.el8 ceph 12 M 2023-09-22T12:50:21.909 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrading: 2023-09-22T12:50:21.909 INFO:teuthology.orchestra.run.smithi096.stdout: librados2 x86_64 2:16.2.14-5.g31c126b7.el8 ceph 3.7 M 2023-09-22T12:50:21.909 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1 x86_64 2:16.2.14-5.g31c126b7.el8 ceph 4.2 M 2023-09-22T12:50:21.909 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2023-09-22T12:50:21.909 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base x86_64 2:16.2.14-5.g31c126b7.el8 ceph 6.1 M 2023-09-22T12:50:21.909 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common x86_64 2:16.2.14-5.g31c126b7.el8 ceph 24 M 2023-09-22T12:50:21.909 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux x86_64 2:16.2.14-5.g31c126b7.el8 ceph 26 k 2023-09-22T12:50:21.909 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-22T12:50:21.909 INFO:teuthology.orchestra.run.smithi096.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-09-22T12:50:21.909 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2 x86_64 2:16.2.14-5.g31c126b7.el8 ceph 824 k 2023-09-22T12:50:21.910 INFO:teuthology.orchestra.run.smithi096.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-22T12:50:21.910 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-22T12:50:21.910 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1 x86_64 2:16.2.14-5.g31c126b7.el8 ceph 494 k 2023-09-22T12:50:21.910 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-22T12:50:21.910 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2 x86_64 2:16.2.14-5.g31c126b7.el8 ceph 3.8 M 2023-09-22T12:50:21.910 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-22T12:50:21.910 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-22T12:50:21.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse x86_64 2:16.2.14-5.g31c126b7.el8 ceph 45 k 2023-09-22T12:50:21.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common x86_64 2:16.2.14-5.g31c126b7.el8 ceph 99 k 2023-09-22T12:50:21.910 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs x86_64 2:16.2.14-5.g31c126b7.el8 ceph 215 k 2023-09-22T12:50:21.911 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-22T12:50:21.911 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados x86_64 2:16.2.14-5.g31c126b7.el8 ceph 390 k 2023-09-22T12:50:21.911 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd x86_64 2:16.2.14-5.g31c126b7.el8 ceph 369 k 2023-09-22T12:50:21.911 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw x86_64 2:16.2.14-5.g31c126b7.el8 ceph 114 k 2023-09-22T12:50:21.911 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:50:21.911 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T12:50:21.911 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:50:21.911 INFO:teuthology.orchestra.run.smithi096.stdout:Install 21 Packages 2023-09-22T12:50:21.911 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrade 2 Packages 2023-09-22T12:50:21.911 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:50:21.912 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 57 M 2023-09-22T12:50:21.912 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T12:50:23.217 INFO:teuthology.orchestra.run.smithi096.stdout:(1/23): ceph-base-16.2.14-5.g31c126b7.el8.x86_6 8.4 MB/s | 6.1 MB 00:00 2023-09-22T12:50:23.267 INFO:teuthology.orchestra.run.smithi096.stdout:(2/23): ceph-selinux-16.2.14-5.g31c126b7.el8.x8 513 kB/s | 26 kB 00:00 2023-09-22T12:50:23.384 INFO:teuthology.orchestra.run.smithi096.stdout:(3/23): libcephfs2-16.2.14-5.g31c126b7.el8.x86_ 6.9 MB/s | 824 kB 00:00 2023-09-22T12:50:23.501 INFO:teuthology.orchestra.run.smithi096.stdout:(4/23): ceph-radosgw-16.2.14-5.g31c126b7.el8.x8 12 MB/s | 12 MB 00:01 2023-09-22T12:50:23.526 INFO:teuthology.orchestra.run.smithi096.stdout:(5/23): libradosstriper1-16.2.14-5.g31c126b7.el 3.4 MB/s | 494 kB 00:00 2023-09-22T12:50:23.594 INFO:teuthology.orchestra.run.smithi096.stdout:(6/23): python3-ceph-argparse-16.2.14-5.g31c126 673 kB/s | 45 kB 00:00 2023-09-22T12:50:23.677 INFO:teuthology.orchestra.run.smithi096.stdout:(7/23): python3-ceph-common-16.2.14-5.g31c126b7 1.2 MB/s | 99 kB 00:00 2023-09-22T12:50:23.744 INFO:teuthology.orchestra.run.smithi096.stdout:(8/23): librgw2-16.2.14-5.g31c126b7.el8.x86_64. 16 MB/s | 3.8 MB 00:00 2023-09-22T12:50:23.769 INFO:teuthology.orchestra.run.smithi096.stdout:(9/23): python3-cephfs-16.2.14-5.g31c126b7.el8. 2.3 MB/s | 215 kB 00:00 2023-09-22T12:50:23.803 INFO:teuthology.orchestra.run.smithi096.stdout:(10/23): python3-rados-16.2.14-5.g31c126b7.el8. 6.5 MB/s | 390 kB 00:00 2023-09-22T12:50:23.828 INFO:teuthology.orchestra.run.smithi096.stdout:(11/23): python3-rbd-16.2.14-5.g31c126b7.el8.x8 6.1 MB/s | 369 kB 00:00 2023-09-22T12:50:23.887 INFO:teuthology.orchestra.run.smithi096.stdout:(12/23): python3-rgw-16.2.14-5.g31c126b7.el8.x8 1.3 MB/s | 114 kB 00:00 2023-09-22T12:50:25.046 INFO:teuthology.orchestra.run.smithi096.stdout:(13/23): ceph-common-16.2.14-5.g31c126b7.el8.x8 9.5 MB/s | 24 MB 00:02 2023-09-22T12:50:26.554 INFO:teuthology.orchestra.run.smithi096.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 29 kB/s | 44 kB 00:01 2023-09-22T12:50:26.862 INFO:teuthology.orchestra.run.smithi096.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 152 kB/s | 47 kB 00:00 2023-09-22T12:50:27.213 INFO:teuthology.orchestra.run.smithi096.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 875 kB/s | 306 kB 00:00 2023-09-22T12:50:27.288 INFO:teuthology.orchestra.run.smithi096.stdout:(17/23): leveldb-1.22-1.el8.x86_64.rpm 2.4 MB/s | 181 kB 00:00 2023-09-22T12:50:27.363 INFO:teuthology.orchestra.run.smithi096.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 783 kB/s | 59 kB 00:00 2023-09-22T12:50:27.438 INFO:teuthology.orchestra.run.smithi096.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 1.0 MB/s | 75 kB 00:00 2023-09-22T12:50:27.789 INFO:teuthology.orchestra.run.smithi096.stdout:(20/23): librados2-16.2.14-5.g31c126b7.el8.x86_ 11 MB/s | 3.7 MB 00:00 2023-09-22T12:50:28.031 INFO:teuthology.orchestra.run.smithi096.stdout:(21/23): librbd1-16.2.14-5.g31c126b7.el8.x86_64 17 MB/s | 4.2 MB 00:00 2023-09-22T12:50:30.414 INFO:teuthology.orchestra.run.smithi096.stdout:(22/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 40 kB/s | 259 kB 00:06 2023-09-22T12:50:35.331 INFO:teuthology.orchestra.run.smithi096.stdout:(23/23): librdkafka-0.11.4-3.el8.x86_64.rpm 31 kB/s | 354 kB 00:11 2023-09-22T12:50:35.334 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T12:50:35.334 INFO:teuthology.orchestra.run.smithi096.stdout:Total 4.3 MB/s | 57 MB 00:13 2023-09-22T12:50:35.374 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T12:50:35.421 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T12:50:35.421 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T12:50:35.777 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T12:50:35.779 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T12:50:36.854 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T12:50:37.054 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-22T12:50:37.232 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-22T12:50:37.599 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-22T12:50:37.799 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librados2-2:16.2.14-5.g31c126b7.el8.x86_64 3/25 2023-09-22T12:50:37.889 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:16.2.14-5.g31c126b7.el8.x86_64 3/25 2023-09-22T12:50:38.168 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rados-2:16.2.14-5.g31c126b7.el8.x86_64 4/25 2023-09-22T12:50:38.373 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs2-2:16.2.14-5.g31c126b7.el8.x86_64 5/25 2023-09-22T12:50:38.777 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs2-2:16.2.14-5.g31c126b7.el8.x86_64 5/25 2023-09-22T12:50:39.098 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librbd1-2:16.2.14-5.g31c126b7.el8.x86_64 6/25 2023-09-22T12:50:39.169 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:16.2.14-5.g31c126b7.el8.x86_64 6/25 2023-09-22T12:50:39.438 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2023-09-22T12:50:39.581 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-09-22T12:50:39.938 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-09-22T12:50:40.098 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librgw2-2:16.2.14-5.g31c126b7.el8.x86_64 9/25 2023-09-22T12:50:40.208 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librgw2-2:16.2.14-5.g31c126b7.el8.x86_64 9/25 2023-09-22T12:50:40.404 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-09-22T12:50:40.561 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-argparse-2:16.2.14-5.g31c126b7.el8.x8 11/25 2023-09-22T12:50:40.739 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cephfs-2:16.2.14-5.g31c126b7.el8.x86_64 12/25 2023-09-22T12:50:40.941 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rgw-2:16.2.14-5.g31c126b7.el8.x86_64 13/25 2023-09-22T12:50:41.140 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rbd-2:16.2.14-5.g31c126b7.el8.x86_64 14/25 2023-09-22T12:50:41.340 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libradosstriper1-2:16.2.14-5.g31c126b7.el8.x86_64 15/25 2023-09-22T12:50:41.398 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libradosstriper1-2:16.2.14-5.g31c126b7.el8.x86_64 15/25 2023-09-22T12:50:41.606 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-22T12:50:41.789 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-22T12:50:41.959 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-22T12:50:42.182 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-common-2:16.2.14-5.g31c126b7.el8.x86_ 19/25 2023-09-22T12:50:44.583 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:16.2.14-5.g31c126b7.el8.x86_64 20/25 2023-09-22T12:50:45.065 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-common-2:16.2.14-5.g31c126b7.el8.x86_64 20/25 2023-09-22T12:50:45.772 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:16.2.14-5.g31c126b7.el8.x86_64 20/25 2023-09-22T12:50:46.111 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-base-2:16.2.14-5.g31c126b7.el8.x86_64 21/25 2023-09-22T12:50:46.201 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-base-2:16.2.14-5.g31c126b7.el8.x86_64 21/25 2023-09-22T12:50:46.322 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-selinux-2:16.2.14-5.g31c126b7.el8.x86_64 22/25 2023-09-22T12:51:04.617 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-selinux-2:16.2.14-5.g31c126b7.el8.x86_64 22/25 2023-09-22T12:51:04.617 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /sys 2023-09-22T12:51:04.617 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /proc 2023-09-22T12:51:04.617 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /dev 2023-09-22T12:51:04.617 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /run 2023-09-22T12:51:04.618 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /mnt 2023-09-22T12:51:04.618 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /var/tmp 2023-09-22T12:51:04.618 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /home 2023-09-22T12:51:04.618 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /tmp 2023-09-22T12:51:04.618 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /dev 2023-09-22T12:51:04.618 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:51:04.995 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 8 - AppStream 30 kB/s | 33 MB 19:13 2023-09-22T12:51:05.782 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-radosgw-2:16.2.14-5.g31c126b7.el8.x86_64 23/25 2023-09-22T12:51:05.854 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-5.g31c126b7.el8.x86_64 23/25 2023-09-22T12:51:05.854 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-22T12:51:06.069 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-22T12:51:06.070 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-22T12:51:06.761 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-22T12:51:06.761 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-base-2:16.2.14-5.g31c126b7.el8.x86_64 1/25 2023-09-22T12:51:06.761 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-common-2:16.2.14-5.g31c126b7.el8.x86_64 2/25 2023-09-22T12:51:06.761 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-radosgw-2:16.2.14-5.g31c126b7.el8.x86_64 3/25 2023-09-22T12:51:06.762 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-selinux-2:16.2.14-5.g31c126b7.el8.x86_64 4/25 2023-09-22T12:51:06.762 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs2-2:16.2.14-5.g31c126b7.el8.x86_64 5/25 2023-09-22T12:51:06.762 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libradosstriper1-2:16.2.14-5.g31c126b7.el8.x86_64 6/25 2023-09-22T12:51:06.762 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librgw2-2:16.2.14-5.g31c126b7.el8.x86_64 7/25 2023-09-22T12:51:06.762 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-argparse-2:16.2.14-5.g31c126b7.el8.x8 8/25 2023-09-22T12:51:06.762 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-common-2:16.2.14-5.g31c126b7.el8.x86_ 9/25 2023-09-22T12:51:06.762 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cephfs-2:16.2.14-5.g31c126b7.el8.x86_64 10/25 2023-09-22T12:51:06.762 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rados-2:16.2.14-5.g31c126b7.el8.x86_64 11/25 2023-09-22T12:51:06.762 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rbd-2:16.2.14-5.g31c126b7.el8.x86_64 12/25 2023-09-22T12:51:06.762 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rgw-2:16.2.14-5.g31c126b7.el8.x86_64 13/25 2023-09-22T12:51:06.762 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-22T12:51:06.763 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-22T12:51:06.763 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-22T12:51:06.763 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-22T12:51:06.763 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-22T12:51:06.763 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2023-09-22T12:51:06.763 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2023-09-22T12:51:06.763 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2023-09-22T12:51:06.763 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:16.2.14-5.g31c126b7.el8.x86_64 22/25 2023-09-22T12:51:06.763 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-22T12:51:06.763 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:16.2.14-5.g31c126b7.el8.x86_64 24/25 2023-09-22T12:51:07.919 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-22T12:51:07.920 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:51:07.920 INFO:teuthology.orchestra.run.smithi096.stdout:Upgraded: 2023-09-22T12:51:07.920 INFO:teuthology.orchestra.run.smithi096.stdout: librados2-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:51:07.920 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:51:07.920 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T12:51:07.920 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:51:07.920 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:51:07.920 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:51:07.920 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:51:07.920 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-22T12:51:07.921 INFO:teuthology.orchestra.run.smithi096.stdout: leveldb-1.22-1.el8.x86_64 2023-09-22T12:51:07.921 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:51:07.921 INFO:teuthology.orchestra.run.smithi096.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-22T12:51:07.921 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-22T12:51:07.921 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:51:07.921 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-22T12:51:07.921 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:51:07.921 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-22T12:51:07.921 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-22T12:51:07.921 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:51:07.921 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:51:07.921 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:51:07.922 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-22T12:51:07.922 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:51:07.922 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:51:07.922 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:51:07.922 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:51:07.922 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T12:51:08.240 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-test 2023-09-22T12:51:09.086 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:52 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T12:51:10.644 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T12:51:10.645 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:51:10.645 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2023-09-22T12:51:10.645 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:51:10.645 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T12:51:10.645 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test x86_64 2:16.2.14-5.g31c126b7.el8 ceph 52 M 2023-09-22T12:51:10.646 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2023-09-22T12:51:10.646 INFO:teuthology.orchestra.run.smithi096.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-22T12:51:10.646 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite x86_64 2:16.2.14-5.g31c126b7.el8 ceph 174 k 2023-09-22T12:51:10.646 INFO:teuthology.orchestra.run.smithi096.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-22T12:51:10.646 INFO:teuthology.orchestra.run.smithi096.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-22T12:51:10.646 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-22T12:51:10.646 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:51:10.646 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T12:51:10.646 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:51:10.646 INFO:teuthology.orchestra.run.smithi096.stdout:Install 6 Packages 2023-09-22T12:51:10.646 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:51:10.647 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 53 M 2023-09-22T12:51:10.647 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 196 M 2023-09-22T12:51:10.647 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T12:51:12.056 INFO:teuthology.orchestra.run.smithi096.stdout:(1/6): libcephsqlite-16.2.14-5.g31c126b7.el8.x8 850 kB/s | 174 kB 00:00 2023-09-22T12:51:13.670 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 8 - BaseOS 18 MB/s | 47 MB 00:02 2023-09-22T12:51:14.372 INFO:teuthology.orchestra.run.smithi096.stdout:(2/6): ceph-test-16.2.14-5.g31c126b7.el8.x86_64 21 MB/s | 52 MB 00:02 2023-09-22T12:51:16.455 INFO:teuthology.orchestra.run.smithi096.stdout:(3/6): jq-1.6-7.el8.x86_64.rpm 44 kB/s | 202 kB 00:04 2023-09-22T12:51:20.729 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 8 - Extras 45 kB/s | 18 kB 00:00 2023-09-22T12:51:21.113 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 8 - Extras common packages 33 kB/s | 6.9 kB 00:00 2023-09-22T12:51:22.497 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 8 - PowerTools 5.2 MB/s | 6.5 MB 00:01 2023-09-22T12:51:23.779 INFO:teuthology.orchestra.run.smithi096.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 16 kB/s | 187 kB 00:11 2023-09-22T12:51:23.915 INFO:teuthology.orchestra.run.smithi105.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-09-22T12:51:24.941 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 18 MB/s | 16 MB 00:00 2023-09-22T12:51:27.786 INFO:teuthology.orchestra.run.smithi096.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 24 kB/s | 323 kB 00:13 2023-09-22T12:51:28.733 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 511 kB/s | 24 kB 00:00 2023-09-22T12:51:33.125 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T12:51:33.128 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:51:33.128 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2023-09-22T12:51:33.128 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:51:33.128 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-09-22T12:51:33.128 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw x86_64 2:16.2.14-5.g31c126b7.el8 ceph 12 M 2023-09-22T12:51:33.128 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrading: 2023-09-22T12:51:33.129 INFO:teuthology.orchestra.run.smithi105.stdout: librados2 x86_64 2:16.2.14-5.g31c126b7.el8 ceph 3.7 M 2023-09-22T12:51:33.129 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1 x86_64 2:16.2.14-5.g31c126b7.el8 ceph 4.2 M 2023-09-22T12:51:33.129 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2023-09-22T12:51:33.129 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base x86_64 2:16.2.14-5.g31c126b7.el8 ceph 6.1 M 2023-09-22T12:51:33.129 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common x86_64 2:16.2.14-5.g31c126b7.el8 ceph 24 M 2023-09-22T12:51:33.129 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux x86_64 2:16.2.14-5.g31c126b7.el8 ceph 26 k 2023-09-22T12:51:33.129 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-22T12:51:33.129 INFO:teuthology.orchestra.run.smithi105.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-09-22T12:51:33.129 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2 x86_64 2:16.2.14-5.g31c126b7.el8 ceph 824 k 2023-09-22T12:51:33.130 INFO:teuthology.orchestra.run.smithi105.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-22T12:51:33.130 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-22T12:51:33.130 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 x86_64 2:16.2.14-5.g31c126b7.el8 ceph 494 k 2023-09-22T12:51:33.130 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-22T12:51:33.130 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2 x86_64 2:16.2.14-5.g31c126b7.el8 ceph 3.8 M 2023-09-22T12:51:33.130 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-22T12:51:33.130 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-22T12:51:33.130 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse x86_64 2:16.2.14-5.g31c126b7.el8 ceph 45 k 2023-09-22T12:51:33.130 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common x86_64 2:16.2.14-5.g31c126b7.el8 ceph 99 k 2023-09-22T12:51:33.130 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs x86_64 2:16.2.14-5.g31c126b7.el8 ceph 215 k 2023-09-22T12:51:33.131 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-22T12:51:33.131 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados x86_64 2:16.2.14-5.g31c126b7.el8 ceph 390 k 2023-09-22T12:51:33.131 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd x86_64 2:16.2.14-5.g31c126b7.el8 ceph 369 k 2023-09-22T12:51:33.131 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw x86_64 2:16.2.14-5.g31c126b7.el8 ceph 114 k 2023-09-22T12:51:33.131 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:51:33.131 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-09-22T12:51:33.131 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:51:33.131 INFO:teuthology.orchestra.run.smithi105.stdout:Install 21 Packages 2023-09-22T12:51:33.131 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrade 2 Packages 2023-09-22T12:51:33.131 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:51:33.132 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 57 M 2023-09-22T12:51:33.132 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-09-22T12:51:33.852 INFO:teuthology.orchestra.run.smithi096.stdout:(6/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 4.1 kB/s | 71 kB 00:17 2023-09-22T12:51:33.853 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T12:51:33.854 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.3 MB/s | 53 MB 00:23 2023-09-22T12:51:33.896 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T12:51:33.916 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T12:51:33.916 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T12:51:34.242 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T12:51:34.245 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T12:51:34.292 INFO:teuthology.orchestra.run.smithi105.stdout:(1/23): ceph-base-16.2.14-5.g31c126b7.el8.x86_6 7.6 MB/s | 6.1 MB 00:00 2023-09-22T12:51:34.342 INFO:teuthology.orchestra.run.smithi105.stdout:(2/23): ceph-selinux-16.2.14-5.g31c126b7.el8.x8 515 kB/s | 26 kB 00:00 2023-09-22T12:51:34.476 INFO:teuthology.orchestra.run.smithi105.stdout:(3/23): libcephfs2-16.2.14-5.g31c126b7.el8.x86_ 6.0 MB/s | 824 kB 00:00 2023-09-22T12:51:34.560 INFO:teuthology.orchestra.run.smithi105.stdout:(4/23): libradosstriper1-16.2.14-5.g31c126b7.el 5.8 MB/s | 494 kB 00:00 2023-09-22T12:51:34.772 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T12:51:34.985 INFO:teuthology.orchestra.run.smithi105.stdout:(5/23): librgw2-16.2.14-5.g31c126b7.el8.x86_64. 8.9 MB/s | 3.8 MB 00:00 2023-09-22T12:51:34.995 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-22T12:51:35.202 INFO:teuthology.orchestra.run.smithi105.stdout:(6/23): ceph-common-16.2.14-5.g31c126b7.el8.x86 14 MB/s | 24 MB 00:01 2023-09-22T12:51:35.219 INFO:teuthology.orchestra.run.smithi105.stdout:(7/23): python3-ceph-argparse-16.2.14-5.g31c126 193 kB/s | 45 kB 00:00 2023-09-22T12:51:35.221 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-22T12:51:35.252 INFO:teuthology.orchestra.run.smithi105.stdout:(8/23): python3-ceph-common-16.2.14-5.g31c126b7 1.9 MB/s | 99 kB 00:00 2023-09-22T12:51:35.278 INFO:teuthology.orchestra.run.smithi105.stdout:(9/23): python3-cephfs-16.2.14-5.g31c126b7.el8. 3.6 MB/s | 215 kB 00:00 2023-09-22T12:51:35.303 INFO:teuthology.orchestra.run.smithi105.stdout:(10/23): python3-rados-16.2.14-5.g31c126b7.el8. 7.6 MB/s | 390 kB 00:00 2023-09-22T12:51:35.328 INFO:teuthology.orchestra.run.smithi105.stdout:(11/23): python3-rbd-16.2.14-5.g31c126b7.el8.x8 7.2 MB/s | 369 kB 00:00 2023-09-22T12:51:35.353 INFO:teuthology.orchestra.run.smithi105.stdout:(12/23): python3-rgw-16.2.14-5.g31c126b7.el8.x8 2.2 MB/s | 114 kB 00:00 2023-09-22T12:51:35.411 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-22T12:51:35.493 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-22T12:51:35.692 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-22T12:51:35.852 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephsqlite-2:16.2.14-5.g31c126b7.el8.x86_64 5/6 2023-09-22T12:51:37.712 INFO:teuthology.orchestra.run.smithi105.stdout:(13/23): ceph-radosgw-16.2.14-5.g31c126b7.el8.x 2.7 MB/s | 12 MB 00:04 2023-09-22T12:51:40.478 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephsqlite-2:16.2.14-5.g31c126b7.el8.x86_64 5/6 2023-09-22T12:51:40.700 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-test-2:16.2.14-5.g31c126b7.el8.x86_64 6/6 2023-09-22T12:51:40.984 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-test-2:16.2.14-5.g31c126b7.el8.x86_64 6/6 2023-09-22T12:51:40.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-test-2:16.2.14-5.g31c126b7.el8.x86_64 1/6 2023-09-22T12:51:40.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephsqlite-2:16.2.14-5.g31c126b7.el8.x86_64 2/6 2023-09-22T12:51:40.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-22T12:51:40.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-22T12:51:40.985 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-22T12:51:41.137 INFO:teuthology.orchestra.run.smithi105.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 45 kB/s | 259 kB 00:05 2023-09-22T12:51:41.438 INFO:teuthology.orchestra.run.smithi105.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 156 kB/s | 47 kB 00:00 2023-09-22T12:51:41.614 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-22T12:51:41.614 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:51:41.614 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T12:51:41.616 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test-2:16.2.14-5.g31c126b7.el8.x86_64 jq-1.6-7.el8.x86_64 2023-09-22T12:51:41.616 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite-2:16.2.14-5.g31c126b7.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-09-22T12:51:41.616 INFO:teuthology.orchestra.run.smithi096.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-09-22T12:51:41.618 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:51:41.619 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T12:51:41.824 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph 2023-09-22T12:51:42.038 INFO:teuthology.orchestra.run.smithi105.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 510 kB/s | 306 kB 00:00 2023-09-22T12:51:42.171 INFO:teuthology.orchestra.run.smithi105.stdout:(17/23): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2023-09-22T12:51:42.280 INFO:teuthology.orchestra.run.smithi105.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 541 kB/s | 59 kB 00:00 2023-09-22T12:51:42.388 INFO:teuthology.orchestra.run.smithi105.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 693 kB/s | 75 kB 00:00 2023-09-22T12:51:42.685 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:25 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T12:51:42.689 INFO:teuthology.orchestra.run.smithi105.stdout:(20/23): librados2-16.2.14-5.g31c126b7.el8.x86_ 12 MB/s | 3.7 MB 00:00 2023-09-22T12:51:42.906 INFO:teuthology.orchestra.run.smithi105.stdout:(21/23): librbd1-16.2.14-5.g31c126b7.el8.x86_64 19 MB/s | 4.2 MB 00:00 2023-09-22T12:51:44.252 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T12:51:44.256 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:51:44.256 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2023-09-22T12:51:44.256 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:51:44.256 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T12:51:44.256 INFO:teuthology.orchestra.run.smithi096.stdout: ceph x86_64 2:16.2.14-5.g31c126b7.el8 ceph 6.4 k 2023-09-22T12:51:44.257 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2023-09-22T12:51:44.257 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds x86_64 2:16.2.14-5.g31c126b7.el8 ceph 2.4 M 2023-09-22T12:51:44.257 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr x86_64 2:16.2.14-5.g31c126b7.el8 ceph 1.6 M 2023-09-22T12:51:44.257 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core noarch 2:16.2.14-5.g31c126b7.el8 ceph-noarch 244 k 2023-09-22T12:51:44.257 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon x86_64 2:16.2.14-5.g31c126b7.el8 ceph 5.0 M 2023-09-22T12:51:44.257 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd x86_64 2:16.2.14-5.g31c126b7.el8 ceph 18 M 2023-09-22T12:51:44.258 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-22T12:51:44.258 INFO:teuthology.orchestra.run.smithi096.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-22T12:51:44.258 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-22T12:51:44.258 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-22T12:51:44.258 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-22T12:51:44.258 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-22T12:51:44.258 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-22T12:51:44.259 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-22T12:51:44.259 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-22T12:51:44.259 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-22T12:51:44.259 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-22T12:51:44.259 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-22T12:51:44.260 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-22T12:51:44.260 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-22T12:51:44.260 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-22T12:51:44.260 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-22T12:51:44.260 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-22T12:51:44.260 INFO:teuthology.orchestra.run.smithi096.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-22T12:51:44.260 INFO:teuthology.orchestra.run.smithi096.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-22T12:51:44.260 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-22T12:51:44.261 INFO:teuthology.orchestra.run.smithi096.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-22T12:51:44.261 INFO:teuthology.orchestra.run.smithi096.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-22T12:51:44.261 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-22T12:51:44.261 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-22T12:51:44.261 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-22T12:51:44.261 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-22T12:51:44.261 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:51:44.261 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T12:51:44.262 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:51:44.262 INFO:teuthology.orchestra.run.smithi096.stdout:Install 32 Packages 2023-09-22T12:51:44.262 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:51:44.263 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 31 M 2023-09-22T12:51:44.263 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 101 M 2023-09-22T12:51:44.263 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T12:51:44.698 INFO:teuthology.orchestra.run.smithi096.stdout:(1/32): ceph-16.2.14-5.g31c126b7.el8.x86_64.rpm 50 kB/s | 6.4 kB 00:00 2023-09-22T12:51:45.098 INFO:teuthology.orchestra.run.smithi105.stdout:(22/23): python3-prettytable-0.7.2-14.el8.noarc 5.9 kB/s | 44 kB 00:07 2023-09-22T12:51:45.139 INFO:teuthology.orchestra.run.smithi096.stdout:(2/32): ceph-mon-16.2.14-5.g31c126b7.el8.x86_64 11 MB/s | 5.0 MB 00:00 2023-09-22T12:51:45.173 INFO:teuthology.orchestra.run.smithi096.stdout:(3/32): ceph-mgr-16.2.14-5.g31c126b7.el8.x86_64 2.7 MB/s | 1.6 MB 00:00 2023-09-22T12:51:45.257 INFO:teuthology.orchestra.run.smithi096.stdout:(4/32): ceph-mds-16.2.14-5.g31c126b7.el8.x86_64 3.6 MB/s | 2.4 MB 00:00 2023-09-22T12:51:45.282 INFO:teuthology.orchestra.run.smithi096.stdout:(5/32): ceph-mgr-modules-core-16.2.14-5.g31c126 2.2 MB/s | 244 kB 00:00 2023-09-22T12:51:46.024 INFO:teuthology.orchestra.run.smithi096.stdout:(6/32): ceph-osd-16.2.14-5.g31c126b7.el8.x86_64 21 MB/s | 18 MB 00:00 2023-09-22T12:51:46.041 INFO:teuthology.orchestra.run.smithi096.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 51 kB/s | 39 kB 00:00 2023-09-22T12:51:47.123 INFO:teuthology.orchestra.run.smithi105.stdout:(23/23): librdkafka-0.11.4-3.el8.x86_64.rpm 30 kB/s | 354 kB 00:11 2023-09-22T12:51:47.125 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-09-22T12:51:47.125 INFO:teuthology.orchestra.run.smithi105.stdout:Total 4.1 MB/s | 57 MB 00:13 2023-09-22T12:51:47.167 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-09-22T12:51:47.213 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-09-22T12:51:47.213 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-09-22T12:51:47.556 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-09-22T12:51:47.559 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-09-22T12:51:47.791 INFO:teuthology.orchestra.run.smithi096.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 59 kB/s | 103 kB 00:01 2023-09-22T12:51:47.816 INFO:teuthology.orchestra.run.smithi096.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 32 kB/s | 57 kB 00:01 2023-09-22T12:51:48.441 INFO:teuthology.orchestra.run.smithi096.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 398 kB/s | 249 kB 00:00 2023-09-22T12:51:48.612 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-09-22T12:51:48.667 INFO:teuthology.orchestra.run.smithi096.stdout:(11/32): python3-cryptography-3.2.1-6.el8.x86_6 2.8 MB/s | 641 kB 00:00 2023-09-22T12:51:48.758 INFO:teuthology.orchestra.run.smithi096.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.2 MB/s | 109 kB 00:00 2023-09-22T12:51:48.822 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-22T12:51:49.015 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-22T12:51:49.100 INFO:teuthology.orchestra.run.smithi096.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 128 kB/s | 44 kB 00:00 2023-09-22T12:51:49.268 INFO:teuthology.orchestra.run.smithi096.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.1 MB/s | 185 kB 00:00 2023-09-22T12:51:49.342 INFO:teuthology.orchestra.run.smithi096.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.3 MB/s | 173 kB 00:00 2023-09-22T12:51:49.380 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-22T12:51:49.485 INFO:teuthology.orchestra.run.smithi096.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.7 MB/s | 384 kB 00:00 2023-09-22T12:51:49.560 INFO:teuthology.orchestra.run.smithi096.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 148 kB/s | 11 kB 00:00 2023-09-22T12:51:49.574 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librados2-2:16.2.14-5.g31c126b7.el8.x86_64 3/25 2023-09-22T12:51:49.627 INFO:teuthology.orchestra.run.smithi096.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 262 kB/s | 18 kB 00:00 2023-09-22T12:51:49.673 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:16.2.14-5.g31c126b7.el8.x86_64 3/25 2023-09-22T12:51:49.702 INFO:teuthology.orchestra.run.smithi096.stdout:(19/32): python3-logutils-0.3.5-11.el8.noarch.r 650 kB/s | 49 kB 00:00 2023-09-22T12:51:49.777 INFO:teuthology.orchestra.run.smithi096.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 783 kB/s | 59 kB 00:00 2023-09-22T12:51:49.894 INFO:teuthology.orchestra.run.smithi096.stdout:(21/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.4 MB/s | 283 kB 00:00 2023-09-22T12:51:49.964 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rados-2:16.2.14-5.g31c126b7.el8.x86_64 4/25 2023-09-22T12:51:49.995 INFO:teuthology.orchestra.run.smithi096.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 156 kB/s | 16 kB 00:00 2023-09-22T12:51:50.070 INFO:teuthology.orchestra.run.smithi096.stdout:(23/32): python3-simplegeneric-0.8.1-17.el8.noa 249 kB/s | 19 kB 00:00 2023-09-22T12:51:50.129 INFO:teuthology.orchestra.run.smithi096.stdout:(24/32): python3-singledispatch-3.4.0.3-18.el8. 410 kB/s | 24 kB 00:00 2023-09-22T12:51:50.157 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs2-2:16.2.14-5.g31c126b7.el8.x86_64 5/25 2023-09-22T12:51:50.187 INFO:teuthology.orchestra.run.smithi096.stdout:(25/32): python3-tempora-1.14.1-5.el8.noarch.rp 502 kB/s | 29 kB 00:00 2023-09-22T12:51:50.246 INFO:teuthology.orchestra.run.smithi096.stdout:(26/32): python3-trustme-0.6.0-4.el8.noarch.rpm 472 kB/s | 27 kB 00:00 2023-09-22T12:51:50.313 INFO:teuthology.orchestra.run.smithi096.stdout:(27/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.7 MB/s | 255 kB 00:00 2023-09-22T12:51:50.380 INFO:teuthology.orchestra.run.smithi096.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.7 MB/s | 251 kB 00:00 2023-09-22T12:51:50.568 INFO:teuthology.orchestra.run.smithi096.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 454 kB/s | 85 kB 00:00 2023-09-22T12:51:50.572 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs2-2:16.2.14-5.g31c126b7.el8.x86_64 5/25 2023-09-22T12:51:50.705 INFO:teuthology.orchestra.run.smithi096.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 167 kB/s | 23 kB 00:00 2023-09-22T12:51:50.774 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librbd1-2:16.2.14-5.g31c126b7.el8.x86_64 6/25 2023-09-22T12:51:50.836 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:16.2.14-5.g31c126b7.el8.x86_64 6/25 2023-09-22T12:51:51.064 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2023-09-22T12:51:51.216 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-09-22T12:51:51.571 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-09-22T12:51:51.932 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librgw2-2:16.2.14-5.g31c126b7.el8.x86_64 9/25 2023-09-22T12:51:52.012 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librgw2-2:16.2.14-5.g31c126b7.el8.x86_64 9/25 2023-09-22T12:51:52.180 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-09-22T12:51:52.355 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-argparse-2:16.2.14-5.g31c126b7.el8.x8 11/25 2023-09-22T12:51:52.538 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cephfs-2:16.2.14-5.g31c126b7.el8.x86_64 12/25 2023-09-22T12:51:52.759 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rgw-2:16.2.14-5.g31c126b7.el8.x86_64 13/25 2023-09-22T12:51:52.805 INFO:teuthology.orchestra.run.smithi096.stdout:(31/32): python3-mako-1.0.6-14.el8.noarch.rpm 21 kB/s | 157 kB 00:07 2023-09-22T12:51:52.966 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rbd-2:16.2.14-5.g31c126b7.el8.x86_64 14/25 2023-09-22T12:51:53.132 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libradosstriper1-2:16.2.14-5.g31c126b7.el8.x86_64 15/25 2023-09-22T12:51:53.208 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libradosstriper1-2:16.2.14-5.g31c126b7.el8.x86_64 15/25 2023-09-22T12:51:53.426 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-22T12:51:53.633 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-22T12:51:53.834 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-22T12:51:54.024 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-common-2:16.2.14-5.g31c126b7.el8.x86_ 19/25 2023-09-22T12:51:56.486 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:16.2.14-5.g31c126b7.el8.x86_64 20/25 2023-09-22T12:51:56.690 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-common-2:16.2.14-5.g31c126b7.el8.x86_64 20/25 2023-09-22T12:51:57.410 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:16.2.14-5.g31c126b7.el8.x86_64 20/25 2023-09-22T12:51:57.697 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-base-2:16.2.14-5.g31c126b7.el8.x86_64 21/25 2023-09-22T12:51:57.787 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-base-2:16.2.14-5.g31c126b7.el8.x86_64 21/25 2023-09-22T12:51:57.915 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-selinux-2:16.2.14-5.g31c126b7.el8.x86_64 22/25 2023-09-22T12:52:10.852 INFO:teuthology.orchestra.run.smithi096.stdout:(32/32): python3-werkzeug-0.12.2-4.el8.noarch.r 20 kB/s | 457 kB 00:23 2023-09-22T12:52:10.855 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T12:52:10.855 INFO:teuthology.orchestra.run.smithi096.stdout:Total 1.2 MB/s | 31 MB 00:26 2023-09-22T12:52:10.941 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T12:52:10.980 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T12:52:10.980 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T12:52:11.326 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T12:52:11.329 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T12:52:13.900 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T12:52:14.166 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-22T12:52:14.393 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-22T12:52:14.593 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-22T12:52:14.867 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-22T12:52:15.132 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-22T12:52:15.313 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-22T12:52:15.497 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-22T12:52:15.700 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-22T12:52:15.798 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-selinux-2:16.2.14-5.g31c126b7.el8.x86_64 22/25 2023-09-22T12:52:15.798 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /sys 2023-09-22T12:52:15.798 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /proc 2023-09-22T12:52:15.799 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /dev 2023-09-22T12:52:15.799 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /run 2023-09-22T12:52:15.799 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /mnt 2023-09-22T12:52:15.799 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /var/tmp 2023-09-22T12:52:15.799 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /home 2023-09-22T12:52:15.799 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /tmp 2023-09-22T12:52:15.799 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /dev 2023-09-22T12:52:15.799 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:52:15.975 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-22T12:52:16.213 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-22T12:52:16.426 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-22T12:52:16.649 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-22T12:52:16.944 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-radosgw-2:16.2.14-5.g31c126b7.el8.x86_64 23/25 2023-09-22T12:52:17.022 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-5.g31c126b7.el8.x86_64 23/25 2023-09-22T12:52:17.022 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-22T12:52:17.082 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-22T12:52:17.247 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-22T12:52:17.247 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-22T12:52:17.346 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-22T12:52:17.573 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-22T12:52:17.810 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-22T12:52:17.923 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-22T12:52:17.924 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-base-2:16.2.14-5.g31c126b7.el8.x86_64 1/25 2023-09-22T12:52:17.924 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-common-2:16.2.14-5.g31c126b7.el8.x86_64 2/25 2023-09-22T12:52:17.924 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-radosgw-2:16.2.14-5.g31c126b7.el8.x86_64 3/25 2023-09-22T12:52:17.924 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-selinux-2:16.2.14-5.g31c126b7.el8.x86_64 4/25 2023-09-22T12:52:17.924 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs2-2:16.2.14-5.g31c126b7.el8.x86_64 5/25 2023-09-22T12:52:17.924 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libradosstriper1-2:16.2.14-5.g31c126b7.el8.x86_64 6/25 2023-09-22T12:52:17.924 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librgw2-2:16.2.14-5.g31c126b7.el8.x86_64 7/25 2023-09-22T12:52:17.924 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-argparse-2:16.2.14-5.g31c126b7.el8.x8 8/25 2023-09-22T12:52:17.925 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-common-2:16.2.14-5.g31c126b7.el8.x86_ 9/25 2023-09-22T12:52:17.925 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cephfs-2:16.2.14-5.g31c126b7.el8.x86_64 10/25 2023-09-22T12:52:17.925 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rados-2:16.2.14-5.g31c126b7.el8.x86_64 11/25 2023-09-22T12:52:17.925 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rbd-2:16.2.14-5.g31c126b7.el8.x86_64 12/25 2023-09-22T12:52:17.925 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rgw-2:16.2.14-5.g31c126b7.el8.x86_64 13/25 2023-09-22T12:52:17.925 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-22T12:52:17.925 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-22T12:52:17.925 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-22T12:52:17.925 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-22T12:52:17.925 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-22T12:52:17.925 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2023-09-22T12:52:17.926 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2023-09-22T12:52:17.926 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2023-09-22T12:52:17.926 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:16.2.14-5.g31c126b7.el8.x86_64 22/25 2023-09-22T12:52:17.926 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-22T12:52:17.926 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:16.2.14-5.g31c126b7.el8.x86_64 24/25 2023-09-22T12:52:18.078 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-22T12:52:18.407 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-22T12:52:18.607 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-22T12:52:18.826 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-22T12:52:18.927 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-22T12:52:18.927 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:52:18.927 INFO:teuthology.orchestra.run.smithi105.stdout:Upgraded: 2023-09-22T12:52:18.927 INFO:teuthology.orchestra.run.smithi105.stdout: librados2-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:18.928 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:18.928 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-09-22T12:52:18.928 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:18.928 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:18.928 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:18.928 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:18.928 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-22T12:52:18.928 INFO:teuthology.orchestra.run.smithi105.stdout: leveldb-1.22-1.el8.x86_64 2023-09-22T12:52:18.928 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:18.928 INFO:teuthology.orchestra.run.smithi105.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-22T12:52:18.928 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-22T12:52:18.928 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:18.928 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-22T12:52:18.929 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:18.929 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-22T12:52:18.929 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-22T12:52:18.929 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:18.929 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:18.929 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:18.929 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-22T12:52:18.929 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:18.929 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:18.929 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:18.929 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:52:18.929 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T12:52:19.032 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-22T12:52:19.203 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-test 2023-09-22T12:52:19.295 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-22T12:52:19.535 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-22T12:52:19.748 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-22T12:52:20.044 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:52 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T12:52:20.085 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-22T12:52:20.404 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-22T12:52:20.741 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-5.g31c126b7.el8.no 27/32 2023-09-22T12:52:20.915 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-2:16.2.14-5.g31c126b7.el8.x86_64 28/32 2023-09-22T12:52:21.599 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T12:52:21.600 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:52:21.600 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2023-09-22T12:52:21.600 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:52:21.601 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-09-22T12:52:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test x86_64 2:16.2.14-5.g31c126b7.el8 ceph 52 M 2023-09-22T12:52:21.601 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2023-09-22T12:52:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-22T12:52:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite x86_64 2:16.2.14-5.g31c126b7.el8 ceph 174 k 2023-09-22T12:52:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-22T12:52:21.601 INFO:teuthology.orchestra.run.smithi105.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-22T12:52:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-22T12:52:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:52:21.602 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-09-22T12:52:21.602 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:52:21.602 INFO:teuthology.orchestra.run.smithi105.stdout:Install 6 Packages 2023-09-22T12:52:21.602 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:52:21.603 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 53 M 2023-09-22T12:52:21.603 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 196 M 2023-09-22T12:52:21.604 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-09-22T12:52:22.097 INFO:teuthology.orchestra.run.smithi105.stdout:(1/6): libcephsqlite-16.2.14-5.g31c126b7.el8.x8 567 kB/s | 174 kB 00:00 2023-09-22T12:52:22.408 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-2:16.2.14-5.g31c126b7.el8.x86_64 28/32 2023-09-22T12:52:22.559 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-osd-2:16.2.14-5.g31c126b7.el8.x86_64 29/32 2023-09-22T12:52:22.993 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-osd-2:16.2.14-5.g31c126b7.el8.x86_64 29/32 2023-09-22T12:52:23.176 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mon-2:16.2.14-5.g31c126b7.el8.x86_64 30/32 2023-09-22T12:52:23.396 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mon-2:16.2.14-5.g31c126b7.el8.x86_64 30/32 2023-09-22T12:52:23.628 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mds-2:16.2.14-5.g31c126b7.el8.x86_64 31/32 2023-09-22T12:52:23.646 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mds-2:16.2.14-5.g31c126b7.el8.x86_64 31/32 2023-09-22T12:52:23.783 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-2:16.2.14-5.g31c126b7.el8.x86_64 32/32 2023-09-22T12:52:24.186 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-2:16.2.14-5.g31c126b7.el8.x86_64 32/32 2023-09-22T12:52:24.187 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-2:16.2.14-5.g31c126b7.el8.x86_64 1/32 2023-09-22T12:52:24.187 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mds-2:16.2.14-5.g31c126b7.el8.x86_64 2/32 2023-09-22T12:52:24.187 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-2:16.2.14-5.g31c126b7.el8.x86_64 3/32 2023-09-22T12:52:24.187 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mon-2:16.2.14-5.g31c126b7.el8.x86_64 4/32 2023-09-22T12:52:24.187 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-osd-2:16.2.14-5.g31c126b7.el8.x86_64 5/32 2023-09-22T12:52:24.187 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-5.g31c126b7.el8.no 6/32 2023-09-22T12:52:24.188 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-22T12:52:24.188 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-22T12:52:24.188 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-22T12:52:24.188 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-22T12:52:24.188 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-22T12:52:24.188 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-22T12:52:24.188 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-22T12:52:24.188 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-22T12:52:24.188 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-22T12:52:24.189 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-22T12:52:24.189 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-22T12:52:24.189 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-22T12:52:24.189 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-22T12:52:24.189 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-22T12:52:24.189 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-22T12:52:24.189 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-22T12:52:24.189 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-22T12:52:24.190 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-22T12:52:24.190 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-22T12:52:24.190 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-22T12:52:24.190 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-22T12:52:24.190 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-22T12:52:24.190 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-22T12:52:24.190 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-22T12:52:24.190 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-22T12:52:24.497 INFO:teuthology.orchestra.run.smithi105.stdout:(2/6): ceph-test-16.2.14-5.g31c126b7.el8.x86_64 19 MB/s | 52 MB 00:02 2023-09-22T12:52:25.243 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-22T12:52:25.243 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:52:25.244 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T12:52:25.244 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:25.244 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:25.244 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:25.244 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core-2:16.2.14-5.g31c126b7.el8.noarch 2023-09-22T12:52:25.244 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:25.244 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:25.244 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-22T12:52:25.245 INFO:teuthology.orchestra.run.smithi096.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-22T12:52:25.245 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-22T12:52:25.245 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-22T12:52:25.245 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-22T12:52:25.245 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-22T12:52:25.245 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-22T12:52:25.245 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-22T12:52:25.245 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-22T12:52:25.245 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-22T12:52:25.246 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-22T12:52:25.246 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-22T12:52:25.246 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-22T12:52:25.246 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend-2.6-1.el8.noarch 2023-09-22T12:52:25.246 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-22T12:52:25.246 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-22T12:52:25.246 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-22T12:52:25.246 INFO:teuthology.orchestra.run.smithi096.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-22T12:52:25.247 INFO:teuthology.orchestra.run.smithi096.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-22T12:52:25.247 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-22T12:52:25.247 INFO:teuthology.orchestra.run.smithi096.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-22T12:52:25.247 INFO:teuthology.orchestra.run.smithi096.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-22T12:52:25.247 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-22T12:52:25.247 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-22T12:52:25.247 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-22T12:52:25.247 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-22T12:52:25.247 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:52:25.248 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T12:52:25.471 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-base 2023-09-22T12:52:25.914 INFO:teuthology.orchestra.run.smithi105.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 49 kB/s | 187 kB 00:03 2023-09-22T12:52:26.322 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:09 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T12:52:27.713 INFO:teuthology.orchestra.run.smithi096.stdout:Package ceph-base-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T12:52:27.895 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T12:52:27.896 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2023-09-22T12:52:27.896 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T12:52:27.984 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install cephadm 2023-09-22T12:52:28.834 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:11 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T12:52:28.855 INFO:teuthology.orchestra.run.smithi105.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 24 kB/s | 71 kB 00:02 2023-09-22T12:52:30.412 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T12:52:30.414 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:52:30.414 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2023-09-22T12:52:30.414 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:52:30.414 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T12:52:30.414 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm noarch 2:16.2.14-5.g31c126b7.el8 ceph-noarch 84 k 2023-09-22T12:52:30.414 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:52:30.414 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T12:52:30.414 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:52:30.415 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2023-09-22T12:52:30.415 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:52:30.415 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 84 k 2023-09-22T12:52:30.415 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 348 k 2023-09-22T12:52:30.415 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T12:52:30.584 INFO:teuthology.orchestra.run.smithi096.stdout:cephadm-16.2.14-5.g31c126b7.el8.noarch.rpm 497 kB/s | 84 kB 00:00 2023-09-22T12:52:30.584 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T12:52:30.585 INFO:teuthology.orchestra.run.smithi096.stdout:Total 493 kB/s | 84 kB 00:00 2023-09-22T12:52:30.585 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T12:52:30.591 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T12:52:30.591 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T12:52:30.593 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T12:52:30.596 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T12:52:30.638 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T12:52:31.081 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:16.2.14-5.g31c126b7.el8.noarch 1/1 2023-09-22T12:52:31.155 INFO:teuthology.orchestra.run.smithi105.stdout:(5/6): jq-1.6-7.el8.x86_64.rpm 22 kB/s | 202 kB 00:09 2023-09-22T12:52:31.250 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephadm-2:16.2.14-5.g31c126b7.el8.noarch 1/1 2023-09-22T12:52:31.478 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:16.2.14-5.g31c126b7.el8.noarch 1/1 2023-09-22T12:52:32.078 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephadm-2:16.2.14-5.g31c126b7.el8.noarch 1/1 2023-09-22T12:52:32.079 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:52:32.079 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T12:52:32.079 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm-2:16.2.14-5.g31c126b7.el8.noarch 2023-09-22T12:52:32.079 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:52:32.079 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T12:52:32.339 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-immutable-object-cache 2023-09-22T12:52:33.187 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:16 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T12:52:34.738 INFO:teuthology.orchestra.run.smithi105.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 32 kB/s | 323 kB 00:10 2023-09-22T12:52:34.739 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-09-22T12:52:34.740 INFO:teuthology.orchestra.run.smithi105.stdout:Total 4.0 MB/s | 53 MB 00:13 2023-09-22T12:52:34.784 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-09-22T12:52:34.795 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T12:52:34.796 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:52:34.796 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2023-09-22T12:52:34.796 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:52:34.796 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T12:52:34.796 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-5.g31c126b7.el8 ceph 161 k 2023-09-22T12:52:34.796 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:52:34.796 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T12:52:34.796 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:52:34.796 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2023-09-22T12:52:34.796 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:52:34.797 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 161 k 2023-09-22T12:52:34.797 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 470 k 2023-09-22T12:52:34.797 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T12:52:34.804 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-09-22T12:52:34.804 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-09-22T12:52:34.972 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-immutable-object-cache-16.2.14-5.g31c126b7 922 kB/s | 161 kB 00:00 2023-09-22T12:52:34.972 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T12:52:34.973 INFO:teuthology.orchestra.run.smithi096.stdout:Total 916 kB/s | 161 kB 00:00 2023-09-22T12:52:34.973 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T12:52:34.983 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T12:52:34.983 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T12:52:35.030 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T12:52:35.033 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T12:52:35.116 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-09-22T12:52:35.119 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-09-22T12:52:35.141 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T12:52:35.306 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-5.g31c126b7.el 1/1 2023-09-22T12:52:35.594 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-09-22T12:52:35.711 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-5.g31c126b7.el 1/1 2023-09-22T12:52:35.826 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-22T12:52:36.036 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-22T12:52:36.151 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-22T12:52:36.235 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-22T12:52:36.307 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-5.g31c126b7.el 1/1 2023-09-22T12:52:36.307 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:52:36.308 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T12:52:36.308 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:52:36.308 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:52:36.308 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T12:52:36.415 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-22T12:52:36.511 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr 2023-09-22T12:52:36.568 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephsqlite-2:16.2.14-5.g31c126b7.el8.x86_64 5/6 2023-09-22T12:52:37.367 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:20 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T12:52:38.772 INFO:teuthology.orchestra.run.smithi096.stdout:Package ceph-mgr-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T12:52:38.959 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T12:52:38.960 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2023-09-22T12:52:38.960 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T12:52:39.067 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-dashboard 2023-09-22T12:52:39.915 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:22 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T12:52:41.176 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephsqlite-2:16.2.14-5.g31c126b7.el8.x86_64 5/6 2023-09-22T12:52:41.390 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-test-2:16.2.14-5.g31c126b7.el8.x86_64 6/6 2023-09-22T12:52:41.477 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T12:52:41.478 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:52:41.478 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2023-09-22T12:52:41.478 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:52:41.479 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T12:52:41.479 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard noarch 2:16.2.14-5.g31c126b7.el8 ceph-noarch 3.1 M 2023-09-22T12:52:41.479 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2023-09-22T12:52:41.479 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards noarch 2:16.2.14-5.g31c126b7.el8 ceph-noarch 24 k 2023-09-22T12:52:41.479 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts noarch 2:16.2.14-5.g31c126b7.el8 ceph-noarch 15 k 2023-09-22T12:52:41.479 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-22T12:52:41.479 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-22T12:52:41.479 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-22T12:52:41.479 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:52:41.479 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T12:52:41.479 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T12:52:41.479 INFO:teuthology.orchestra.run.smithi096.stdout:Install 6 Packages 2023-09-22T12:52:41.480 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:52:41.480 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 3.4 M 2023-09-22T12:52:41.480 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 22 M 2023-09-22T12:52:41.480 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T12:52:41.662 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-test-2:16.2.14-5.g31c126b7.el8.x86_64 6/6 2023-09-22T12:52:41.663 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-test-2:16.2.14-5.g31c126b7.el8.x86_64 1/6 2023-09-22T12:52:41.663 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephsqlite-2:16.2.14-5.g31c126b7.el8.x86_64 2/6 2023-09-22T12:52:41.663 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-22T12:52:41.663 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-22T12:52:41.663 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-22T12:52:42.142 INFO:teuthology.orchestra.run.smithi096.stdout:(1/6): ceph-grafana-dashboards-16.2.14-5.g31c12 87 kB/s | 24 kB 00:00 2023-09-22T12:52:42.167 INFO:teuthology.orchestra.run.smithi096.stdout:(2/6): ceph-prometheus-alerts-16.2.14-5.g31c126 48 kB/s | 15 kB 00:00 2023-09-22T12:52:42.276 INFO:teuthology.orchestra.run.smithi096.stdout:(3/6): ceph-mgr-dashboard-16.2.14-5.g31c126b7.e 7.5 MB/s | 3.1 MB 00:00 2023-09-22T12:52:42.319 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-22T12:52:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:52:42.320 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-09-22T12:52:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test-2:16.2.14-5.g31c126b7.el8.x86_64 jq-1.6-7.el8.x86_64 2023-09-22T12:52:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite-2:16.2.14-5.g31c126b7.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-09-22T12:52:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-09-22T12:52:42.320 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:52:42.320 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T12:52:42.501 INFO:teuthology.orchestra.run.smithi096.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 101 kB/s | 34 kB 00:00 2023-09-22T12:52:42.518 INFO:teuthology.orchestra.run.smithi096.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 113 kB/s | 43 kB 00:00 2023-09-22T12:52:42.553 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph 2023-09-22T12:52:42.743 INFO:teuthology.orchestra.run.smithi096.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 421 kB/s | 196 kB 00:00 2023-09-22T12:52:42.746 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T12:52:42.746 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.7 MB/s | 3.4 MB 00:01 2023-09-22T12:52:42.758 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T12:52:42.769 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T12:52:42.769 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T12:52:42.951 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T12:52:42.954 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T12:52:43.214 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T12:52:43.407 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:15 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T12:52:43.410 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-22T12:52:43.630 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-22T12:52:43.898 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-22T12:52:44.027 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-5.g31c126b7.el8.noa 4/6 2023-09-22T12:52:44.990 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T12:52:44.994 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:52:44.994 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2023-09-22T12:52:44.994 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:52:44.994 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-09-22T12:52:44.994 INFO:teuthology.orchestra.run.smithi105.stdout: ceph x86_64 2:16.2.14-5.g31c126b7.el8 ceph 6.4 k 2023-09-22T12:52:44.994 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2023-09-22T12:52:44.994 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds x86_64 2:16.2.14-5.g31c126b7.el8 ceph 2.4 M 2023-09-22T12:52:44.994 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr x86_64 2:16.2.14-5.g31c126b7.el8 ceph 1.6 M 2023-09-22T12:52:44.995 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core noarch 2:16.2.14-5.g31c126b7.el8 ceph-noarch 244 k 2023-09-22T12:52:44.995 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon x86_64 2:16.2.14-5.g31c126b7.el8 ceph 5.0 M 2023-09-22T12:52:44.995 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd x86_64 2:16.2.14-5.g31c126b7.el8 ceph 18 M 2023-09-22T12:52:44.995 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-22T12:52:44.995 INFO:teuthology.orchestra.run.smithi105.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-22T12:52:44.995 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-22T12:52:44.995 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-22T12:52:44.995 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-22T12:52:44.995 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-22T12:52:44.995 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-22T12:52:44.995 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-22T12:52:44.996 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-22T12:52:44.996 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-22T12:52:44.996 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-22T12:52:44.996 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-22T12:52:44.996 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-22T12:52:44.996 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-22T12:52:44.996 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-22T12:52:44.996 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-22T12:52:44.996 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-22T12:52:44.996 INFO:teuthology.orchestra.run.smithi105.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-22T12:52:44.996 INFO:teuthology.orchestra.run.smithi105.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-22T12:52:44.996 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-22T12:52:44.997 INFO:teuthology.orchestra.run.smithi105.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-22T12:52:44.997 INFO:teuthology.orchestra.run.smithi105.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-22T12:52:44.997 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-22T12:52:44.997 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-22T12:52:44.997 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-22T12:52:44.997 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-22T12:52:44.997 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:52:44.997 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-09-22T12:52:44.997 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:52:44.997 INFO:teuthology.orchestra.run.smithi105.stdout:Install 32 Packages 2023-09-22T12:52:44.997 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:52:44.998 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 31 M 2023-09-22T12:52:44.998 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 101 M 2023-09-22T12:52:44.998 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-09-22T12:52:45.026 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-5.g31c126b7.el8.no 5/6 2023-09-22T12:52:45.284 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-5.g31c126b7.el8.noarch 6/6 2023-09-22T12:52:45.298 INFO:teuthology.orchestra.run.smithi105.stdout:(1/32): ceph-16.2.14-5.g31c126b7.el8.x86_64.rpm 35 kB/s | 6.4 kB 00:00 2023-09-22T12:52:45.563 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-5.g31c126b7.el8.noarch 6/6 2023-09-22T12:52:45.563 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-5.g31c126b7.el8.no 1/6 2023-09-22T12:52:45.563 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-5.g31c126b7.el8.noarch 2/6 2023-09-22T12:52:45.563 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-5.g31c126b7.el8.noa 3/6 2023-09-22T12:52:45.563 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-22T12:52:45.563 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-22T12:52:45.632 INFO:teuthology.orchestra.run.smithi105.stdout:(2/32): ceph-mds-16.2.14-5.g31c126b7.el8.x86_64 4.7 MB/s | 2.4 MB 00:00 2023-09-22T12:52:45.666 INFO:teuthology.orchestra.run.smithi105.stdout:(3/32): ceph-mgr-16.2.14-5.g31c126b7.el8.x86_64 2.9 MB/s | 1.6 MB 00:00 2023-09-22T12:52:45.758 INFO:teuthology.orchestra.run.smithi105.stdout:(4/32): ceph-mgr-modules-core-16.2.14-5.g31c126 2.6 MB/s | 244 kB 00:00 2023-09-22T12:52:46.182 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-22T12:52:46.182 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:52:46.182 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T12:52:46.182 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards-2:16.2.14-5.g31c126b7.el8.noarch 2023-09-22T12:52:46.182 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard-2:16.2.14-5.g31c126b7.el8.noarch 2023-09-22T12:52:46.182 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts-2:16.2.14-5.g31c126b7.el8.noarch 2023-09-22T12:52:46.182 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-22T12:52:46.183 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-22T12:52:46.183 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-22T12:52:46.183 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:52:46.183 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T12:52:46.258 INFO:teuthology.orchestra.run.smithi105.stdout:(5/32): ceph-mon-16.2.14-5.g31c126b7.el8.x86_64 5.2 MB/s | 5.0 MB 00:00 2023-09-22T12:52:46.303 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-22T12:52:46.625 INFO:teuthology.orchestra.run.smithi105.stdout:(6/32): ceph-osd-16.2.14-5.g31c126b7.el8.x86_64 18 MB/s | 18 MB 00:00 2023-09-22T12:52:46.650 INFO:teuthology.orchestra.run.smithi105.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 99 kB/s | 39 kB 00:00 2023-09-22T12:52:47.151 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:30 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T12:52:48.576 INFO:teuthology.orchestra.run.smithi105.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 30 kB/s | 57 kB 00:01 2023-09-22T12:52:48.727 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T12:52:48.728 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================================ 2023-09-22T12:52:48.728 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2023-09-22T12:52:48.728 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================================ 2023-09-22T12:52:48.728 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T12:52:48.728 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-5.g31c126b7.el8 ceph-noarch 8.6 M 2023-09-22T12:52:48.728 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2023-09-22T12:52:48.729 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-22T12:52:48.729 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-22T12:52:48.729 INFO:teuthology.orchestra.run.smithi096.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-22T12:52:48.729 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-22T12:52:48.729 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-22T12:52:48.729 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-22T12:52:48.729 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-22T12:52:48.729 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:52:48.729 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T12:52:48.729 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================================ 2023-09-22T12:52:48.729 INFO:teuthology.orchestra.run.smithi096.stdout:Install 8 Packages 2023-09-22T12:52:48.729 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T12:52:48.730 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 37 M 2023-09-22T12:52:48.730 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 210 M 2023-09-22T12:52:48.730 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T12:52:49.441 INFO:teuthology.orchestra.run.smithi096.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.14-5. 15 MB/s | 8.6 MB 00:00 2023-09-22T12:52:49.459 INFO:teuthology.orchestra.run.smithi105.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 36 kB/s | 103 kB 00:02 2023-09-22T12:52:49.909 INFO:teuthology.orchestra.run.smithi105.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 553 kB/s | 249 kB 00:00 2023-09-22T12:52:50.076 INFO:teuthology.orchestra.run.smithi105.stdout:(11/32): python3-cryptography-3.2.1-6.el8.x86_6 3.8 MB/s | 641 kB 00:00 2023-09-22T12:52:50.168 INFO:teuthology.orchestra.run.smithi105.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.2 MB/s | 109 kB 00:00 2023-09-22T12:52:50.927 INFO:teuthology.orchestra.run.smithi105.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 57 kB/s | 44 kB 00:00 2023-09-22T12:52:51.677 INFO:teuthology.orchestra.run.smithi105.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 247 kB/s | 185 kB 00:00 2023-09-22T12:52:52.436 INFO:teuthology.orchestra.run.smithi105.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 228 kB/s | 173 kB 00:00 2023-09-22T12:52:53.228 INFO:teuthology.orchestra.run.smithi105.stdout:(16/32): python3-mako-1.0.6-14.el8.noarch.rpm 21 kB/s | 157 kB 00:07 2023-09-22T12:52:53.345 INFO:teuthology.orchestra.run.smithi105.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 423 kB/s | 384 kB 00:00 2023-09-22T12:52:53.578 INFO:teuthology.orchestra.run.smithi105.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 32 kB/s | 11 kB 00:00 2023-09-22T12:52:53.712 INFO:teuthology.orchestra.run.smithi105.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 48 kB/s | 18 kB 00:00 2023-09-22T12:52:54.095 INFO:teuthology.orchestra.run.smithi105.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 94 kB/s | 49 kB 00:00 2023-09-22T12:52:54.146 INFO:teuthology.orchestra.run.smithi105.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 135 kB/s | 59 kB 00:00 2023-09-22T12:52:54.512 INFO:teuthology.orchestra.run.smithi105.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 42 kB/s | 16 kB 00:00 2023-09-22T12:52:54.846 INFO:teuthology.orchestra.run.smithi105.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 377 kB/s | 283 kB 00:00 2023-09-22T12:52:54.863 INFO:teuthology.orchestra.run.smithi105.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 53 kB/s | 19 kB 00:00 2023-09-22T12:52:55.197 INFO:teuthology.orchestra.run.smithi105.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 68 kB/s | 24 kB 00:00 2023-09-22T12:52:55.289 INFO:teuthology.orchestra.run.smithi105.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 69 kB/s | 29 kB 00:00 2023-09-22T12:52:55.556 INFO:teuthology.orchestra.run.smithi105.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 77 kB/s | 27 kB 00:00 2023-09-22T12:52:55.964 INFO:teuthology.orchestra.run.smithi105.stdout:(28/32): python3-waitress-1.4.3-1.el8.noarch.rp 381 kB/s | 255 kB 00:00 2023-09-22T12:52:56.148 INFO:teuthology.orchestra.run.smithi105.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 424 kB/s | 251 kB 00:00 2023-09-22T12:52:56.507 INFO:teuthology.orchestra.run.smithi105.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 157 kB/s | 85 kB 00:00 2023-09-22T12:52:56.581 INFO:teuthology.orchestra.run.smithi105.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 53 kB/s | 23 kB 00:00 2023-09-22T12:53:34.779 INFO:teuthology.orchestra.run.smithi105.stdout:(32/32): python3-werkzeug-0.12.2-4.el8.noarch.r 9.9 kB/s | 457 kB 00:46 2023-09-22T12:53:34.781 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-09-22T12:53:34.781 INFO:teuthology.orchestra.run.smithi105.stdout:Total 644 kB/s | 31 MB 00:49 2023-09-22T12:53:34.861 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-09-22T12:53:34.900 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-09-22T12:53:34.900 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-09-22T12:53:35.218 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-09-22T12:53:35.221 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-09-22T12:53:36.298 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-09-22T12:53:36.499 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-22T12:53:36.718 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-22T12:53:36.935 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-22T12:53:37.174 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-22T12:53:37.364 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-22T12:53:37.553 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-22T12:53:37.746 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-22T12:53:37.956 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-22T12:53:38.166 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-22T12:53:38.428 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-22T12:53:38.651 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-22T12:53:38.885 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-22T12:53:39.267 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-22T12:53:39.554 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-22T12:53:39.782 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-22T12:53:40.027 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-22T12:53:40.260 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-22T12:53:40.573 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-22T12:53:40.798 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-22T12:53:41.020 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-22T12:53:41.206 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-22T12:53:41.458 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-22T12:53:41.675 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-22T12:53:41.886 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-22T12:53:42.209 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-22T12:53:42.504 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-22T12:53:42.817 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-5.g31c126b7.el8.no 27/32 2023-09-22T12:53:42.983 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-2:16.2.14-5.g31c126b7.el8.x86_64 28/32 2023-09-22T12:53:44.473 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-2:16.2.14-5.g31c126b7.el8.x86_64 28/32 2023-09-22T12:53:44.642 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-osd-2:16.2.14-5.g31c126b7.el8.x86_64 29/32 2023-09-22T12:53:45.067 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-osd-2:16.2.14-5.g31c126b7.el8.x86_64 29/32 2023-09-22T12:53:45.226 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mon-2:16.2.14-5.g31c126b7.el8.x86_64 30/32 2023-09-22T12:53:45.442 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mon-2:16.2.14-5.g31c126b7.el8.x86_64 30/32 2023-09-22T12:53:45.632 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mds-2:16.2.14-5.g31c126b7.el8.x86_64 31/32 2023-09-22T12:53:45.649 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mds-2:16.2.14-5.g31c126b7.el8.x86_64 31/32 2023-09-22T12:53:45.750 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-2:16.2.14-5.g31c126b7.el8.x86_64 32/32 2023-09-22T12:53:46.146 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-2:16.2.14-5.g31c126b7.el8.x86_64 32/32 2023-09-22T12:53:46.146 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-2:16.2.14-5.g31c126b7.el8.x86_64 1/32 2023-09-22T12:53:46.146 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mds-2:16.2.14-5.g31c126b7.el8.x86_64 2/32 2023-09-22T12:53:46.146 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-2:16.2.14-5.g31c126b7.el8.x86_64 3/32 2023-09-22T12:53:46.146 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mon-2:16.2.14-5.g31c126b7.el8.x86_64 4/32 2023-09-22T12:53:46.147 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-osd-2:16.2.14-5.g31c126b7.el8.x86_64 5/32 2023-09-22T12:53:46.147 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-5.g31c126b7.el8.no 6/32 2023-09-22T12:53:46.147 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-22T12:53:46.147 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-22T12:53:46.147 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-22T12:53:46.147 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-22T12:53:46.147 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-22T12:53:46.147 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-22T12:53:46.147 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-22T12:53:46.147 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-22T12:53:46.148 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-22T12:53:46.148 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-22T12:53:46.148 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-22T12:53:46.148 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-22T12:53:46.148 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-22T12:53:46.148 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-22T12:53:46.148 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-22T12:53:46.148 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-22T12:53:46.148 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-22T12:53:46.148 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-22T12:53:46.149 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-22T12:53:46.149 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-22T12:53:46.149 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-22T12:53:46.149 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-22T12:53:46.149 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-22T12:53:46.149 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-22T12:53:46.149 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-22T12:53:47.151 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-22T12:53:47.466 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:53:47.466 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-09-22T12:53:47.466 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:53:47.466 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:53:47.467 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:53:47.467 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core-2:16.2.14-5.g31c126b7.el8.noarch 2023-09-22T12:53:47.467 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:53:47.467 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:53:47.467 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-22T12:53:47.467 INFO:teuthology.orchestra.run.smithi105.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-22T12:53:47.467 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-22T12:53:47.467 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-22T12:53:47.467 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-22T12:53:47.467 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-22T12:53:47.467 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-22T12:53:47.467 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-22T12:53:47.468 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-22T12:53:47.468 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-22T12:53:47.468 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-22T12:53:47.468 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-22T12:53:47.468 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-22T12:53:47.468 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend-2.6-1.el8.noarch 2023-09-22T12:53:47.468 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-22T12:53:47.468 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-22T12:53:47.468 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-22T12:53:47.468 INFO:teuthology.orchestra.run.smithi105.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-22T12:53:47.468 INFO:teuthology.orchestra.run.smithi105.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-22T12:53:47.468 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-22T12:53:47.468 INFO:teuthology.orchestra.run.smithi105.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-22T12:53:47.469 INFO:teuthology.orchestra.run.smithi105.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-22T12:53:47.469 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-22T12:53:47.469 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-22T12:53:47.469 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-22T12:53:47.469 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-22T12:53:47.469 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:53:47.469 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T12:53:47.470 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-base 2023-09-22T12:53:48.325 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:20 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T12:53:49.722 INFO:teuthology.orchestra.run.smithi105.stdout:Package ceph-base-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T12:53:49.907 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T12:53:49.908 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2023-09-22T12:53:49.908 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T12:53:50.023 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install cephadm 2023-09-22T12:53:50.868 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:22 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T12:53:52.447 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T12:53:52.448 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:53:52.448 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2023-09-22T12:53:52.448 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:53:52.448 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-09-22T12:53:52.448 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm noarch 2:16.2.14-5.g31c126b7.el8 ceph-noarch 84 k 2023-09-22T12:53:52.448 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:53:52.448 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-09-22T12:53:52.448 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:53:52.448 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2023-09-22T12:53:52.449 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:53:52.449 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 84 k 2023-09-22T12:53:52.449 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 348 k 2023-09-22T12:53:52.449 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-09-22T12:53:52.620 INFO:teuthology.orchestra.run.smithi105.stdout:cephadm-16.2.14-5.g31c126b7.el8.noarch.rpm 491 kB/s | 84 kB 00:00 2023-09-22T12:53:52.621 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-09-22T12:53:52.621 INFO:teuthology.orchestra.run.smithi105.stdout:Total 488 kB/s | 84 kB 00:00 2023-09-22T12:53:52.622 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-09-22T12:53:52.628 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-09-22T12:53:52.628 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-09-22T12:53:52.630 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-09-22T12:53:52.633 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-09-22T12:53:52.680 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-09-22T12:53:53.136 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:16.2.14-5.g31c126b7.el8.noarch 1/1 2023-09-22T12:53:53.302 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephadm-2:16.2.14-5.g31c126b7.el8.noarch 1/1 2023-09-22T12:53:53.531 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:16.2.14-5.g31c126b7.el8.noarch 1/1 2023-09-22T12:53:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephadm-2:16.2.14-5.g31c126b7.el8.noarch 1/1 2023-09-22T12:53:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:53:54.159 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-09-22T12:53:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm-2:16.2.14-5.g31c126b7.el8.noarch 2023-09-22T12:53:54.159 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:53:54.160 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T12:53:54.486 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-immutable-object-cache 2023-09-22T12:53:55.339 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:27 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T12:53:56.913 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T12:53:56.914 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:53:56.914 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2023-09-22T12:53:56.914 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:53:56.914 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-09-22T12:53:56.914 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-5.g31c126b7.el8 ceph 161 k 2023-09-22T12:53:56.915 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:53:56.915 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-09-22T12:53:56.915 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:53:56.915 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2023-09-22T12:53:56.915 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:53:56.915 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 161 k 2023-09-22T12:53:56.916 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 470 k 2023-09-22T12:53:56.916 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-09-22T12:53:57.102 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-immutable-object-cache-16.2.14-5.g31c126b7 871 kB/s | 161 kB 00:00 2023-09-22T12:53:57.102 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-09-22T12:53:57.102 INFO:teuthology.orchestra.run.smithi105.stdout:Total 865 kB/s | 161 kB 00:00 2023-09-22T12:53:57.102 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-09-22T12:53:57.112 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-09-22T12:53:57.112 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-09-22T12:53:57.159 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-09-22T12:53:57.162 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-09-22T12:53:57.268 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-09-22T12:53:57.440 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-5.g31c126b7.el 1/1 2023-09-22T12:53:57.857 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-5.g31c126b7.el 1/1 2023-09-22T12:53:58.492 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-5.g31c126b7.el 1/1 2023-09-22T12:53:58.492 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:53:58.492 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-09-22T12:53:58.493 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T12:53:58.493 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:53:58.493 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T12:53:58.704 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr 2023-09-22T12:53:59.561 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:31 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T12:54:00.953 INFO:teuthology.orchestra.run.smithi105.stdout:Package ceph-mgr-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T12:54:01.134 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T12:54:01.136 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2023-09-22T12:54:01.136 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T12:54:01.243 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-dashboard 2023-09-22T12:54:02.096 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:34 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T12:54:03.665 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T12:54:03.666 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:54:03.666 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2023-09-22T12:54:03.666 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:54:03.666 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-09-22T12:54:03.667 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard noarch 2:16.2.14-5.g31c126b7.el8 ceph-noarch 3.1 M 2023-09-22T12:54:03.667 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2023-09-22T12:54:03.667 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards noarch 2:16.2.14-5.g31c126b7.el8 ceph-noarch 24 k 2023-09-22T12:54:03.667 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts noarch 2:16.2.14-5.g31c126b7.el8 ceph-noarch 15 k 2023-09-22T12:54:03.667 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-22T12:54:03.667 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-22T12:54:03.667 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-22T12:54:03.667 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:54:03.668 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-09-22T12:54:03.668 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T12:54:03.668 INFO:teuthology.orchestra.run.smithi105.stdout:Install 6 Packages 2023-09-22T12:54:03.668 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:54:03.669 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 3.4 M 2023-09-22T12:54:03.669 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 22 M 2023-09-22T12:54:03.669 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-09-22T12:54:05.563 INFO:teuthology.orchestra.run.smithi105.stdout:(1/6): ceph-prometheus-alerts-16.2.14-5.g31c126 105 kB/s | 15 kB 00:00 2023-09-22T12:54:05.663 INFO:teuthology.orchestra.run.smithi105.stdout:(2/6): ceph-grafana-dashboards-16.2.14-5.g31c12 101 kB/s | 24 kB 00:00 2023-09-22T12:54:05.763 INFO:teuthology.orchestra.run.smithi105.stdout:(3/6): ceph-mgr-dashboard-16.2.14-5.g31c126b7.e 9.1 MB/s | 3.1 MB 00:00 2023-09-22T12:54:05.805 INFO:teuthology.orchestra.run.smithi105.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 237 kB/s | 34 kB 00:00 2023-09-22T12:54:05.838 INFO:teuthology.orchestra.run.smithi105.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 154 kB/s | 43 kB 00:00 2023-09-22T12:54:05.989 INFO:teuthology.orchestra.run.smithi105.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 872 kB/s | 196 kB 00:00 2023-09-22T12:54:05.991 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-09-22T12:54:05.991 INFO:teuthology.orchestra.run.smithi105.stdout:Total 1.5 MB/s | 3.4 MB 00:02 2023-09-22T12:54:06.003 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-09-22T12:54:06.014 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-09-22T12:54:06.014 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-09-22T12:54:06.180 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-09-22T12:54:06.183 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-09-22T12:54:06.447 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-09-22T12:54:06.620 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-22T12:54:06.840 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-22T12:54:07.025 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-22T12:54:07.137 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-5.g31c126b7.el8.noa 4/6 2023-09-22T12:54:08.136 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-5.g31c126b7.el8.no 5/6 2023-09-22T12:54:08.370 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-5.g31c126b7.el8.noarch 6/6 2023-09-22T12:54:08.660 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-5.g31c126b7.el8.noarch 6/6 2023-09-22T12:54:08.660 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-5.g31c126b7.el8.no 1/6 2023-09-22T12:54:08.660 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-5.g31c126b7.el8.noarch 2/6 2023-09-22T12:54:08.660 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-5.g31c126b7.el8.noa 3/6 2023-09-22T12:54:08.660 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-22T12:54:08.660 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-22T12:54:09.280 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-22T12:54:09.281 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:54:09.281 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-09-22T12:54:09.281 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards-2:16.2.14-5.g31c126b7.el8.noarch 2023-09-22T12:54:09.282 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard-2:16.2.14-5.g31c126b7.el8.noarch 2023-09-22T12:54:09.282 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts-2:16.2.14-5.g31c126b7.el8.noarch 2023-09-22T12:54:09.282 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-22T12:54:09.282 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-22T12:54:09.282 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-22T12:54:09.282 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:54:09.282 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T12:54:09.416 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-22T12:54:10.266 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:42 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T12:54:11.852 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T12:54:11.853 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================================ 2023-09-22T12:54:11.853 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2023-09-22T12:54:11.853 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================================ 2023-09-22T12:54:11.854 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-09-22T12:54:11.854 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-5.g31c126b7.el8 ceph-noarch 8.6 M 2023-09-22T12:54:11.854 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2023-09-22T12:54:11.854 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-22T12:54:11.854 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-22T12:54:11.854 INFO:teuthology.orchestra.run.smithi105.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-22T12:54:11.855 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-22T12:54:11.855 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-22T12:54:11.855 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-22T12:54:11.855 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-22T12:54:11.855 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:54:11.855 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-09-22T12:54:11.855 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================================ 2023-09-22T12:54:11.855 INFO:teuthology.orchestra.run.smithi105.stdout:Install 8 Packages 2023-09-22T12:54:11.855 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T12:54:11.856 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 37 M 2023-09-22T12:54:11.856 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 210 M 2023-09-22T12:54:11.856 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-09-22T12:54:13.477 INFO:teuthology.orchestra.run.smithi105.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.14-5. 6.9 MB/s | 8.6 MB 00:01 2023-09-22T12:55:36.579 INFO:teuthology.orchestra.run.smithi096.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 23 kB/s | 3.7 MB 02:47 2023-09-22T12:55:46.086 INFO:teuthology.orchestra.run.smithi096.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 24 kB/s | 225 kB 00:09 2023-09-22T12:56:43.800 INFO:teuthology.orchestra.run.smithi096.stdout:(4/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 21 kB/s | 4.9 MB 03:54 2023-09-22T12:56:44.308 INFO:teuthology.orchestra.run.smithi096.stdout:(5/8): libgfortran-8.5.0-20.el8.x86_64.rpm 1.2 MB/s | 645 kB 00:00 2023-09-22T12:56:44.400 INFO:teuthology.orchestra.run.smithi096.stdout:(6/8): libquadmath-8.5.0-20.el8.x86_64.rpm 1.8 MB/s | 172 kB 00:00 2023-09-22T12:56:54.916 INFO:teuthology.orchestra.run.smithi096.stdout:(7/8): openblas-0.3.15-4.el8.x86_64.rpm 20 kB/s | 4.7 MB 04:06 2023-09-22T12:57:17.011 INFO:teuthology.orchestra.run.smithi105.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 21 kB/s | 3.7 MB 03:03 2023-09-22T12:57:30.101 INFO:teuthology.orchestra.run.smithi105.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 17 kB/s | 225 kB 00:13 2023-09-22T12:57:55.683 INFO:teuthology.orchestra.run.smithi105.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 22 kB/s | 4.7 MB 03:43 2023-09-22T12:57:56.166 INFO:teuthology.orchestra.run.smithi105.stdout:(5/8): libgfortran-8.5.0-20.el8.x86_64.rpm 1.3 MB/s | 645 kB 00:00 2023-09-22T12:57:56.258 INFO:teuthology.orchestra.run.smithi105.stdout:(6/8): libquadmath-8.5.0-20.el8.x86_64.rpm 1.8 MB/s | 172 kB 00:00 2023-09-22T12:57:59.941 INFO:teuthology.orchestra.run.smithi105.stdout:(7/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 22 kB/s | 4.9 MB 03:47 2023-09-22T13:08:08.478 INFO:teuthology.orchestra.run.smithi096.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 19 kB/s | 14 MB 12:22 2023-09-22T13:08:08.479 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:08:08.479 INFO:teuthology.orchestra.run.smithi096.stdout:Total 41 kB/s | 37 MB 15:19 2023-09-22T13:08:08.705 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T13:08:08.724 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T13:08:08.724 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T13:08:09.051 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T13:08:09.054 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T13:08:09.617 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T13:08:09.782 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-22T13:08:09.896 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-22T13:08:10.048 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-22T13:08:10.625 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-22T13:08:10.815 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-22T13:08:11.415 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-22T13:08:11.590 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-22T13:08:12.292 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-22T13:08:12.634 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-22T13:08:12.856 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-22T13:08:14.614 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-22T13:08:15.904 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-22T13:08:16.055 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-5.g31c126b7. 8/8 2023-09-22T13:08:16.368 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-5.g31c126b7. 8/8 2023-09-22T13:08:16.368 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-5.g31c126b7. 1/8 2023-09-22T13:08:16.368 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-22T13:08:16.368 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-22T13:08:16.368 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-22T13:08:16.369 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-22T13:08:16.369 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-22T13:08:16.369 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-22T13:08:16.999 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-22T13:08:17.000 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:08:17.000 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T13:08:17.000 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local-2:16.2.14-5.g31c126b7.el8.noarch 2023-09-22T13:08:17.000 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-22T13:08:17.000 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-22T13:08:17.000 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-22T13:08:17.001 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-22T13:08:17.001 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-22T13:08:17.001 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-22T13:08:17.001 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-22T13:08:17.001 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:08:17.001 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:08:17.103 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-rook 2023-09-22T13:08:17.956 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:18:00 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T13:08:19.531 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T13:08:19.533 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:08:19.533 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2023-09-22T13:08:19.533 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:08:19.533 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T13:08:19.533 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook noarch 2:16.2.14-5.g31c126b7.el8 ceph-noarch 27 k 2023-09-22T13:08:19.534 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2023-09-22T13:08:19.534 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-22T13:08:19.534 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-22T13:08:19.534 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-22T13:08:19.534 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-22T13:08:19.534 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-22T13:08:19.534 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-22T13:08:19.535 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-22T13:08:19.535 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-22T13:08:19.535 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-22T13:08:19.535 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-22T13:08:19.535 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-22T13:08:19.535 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-22T13:08:19.535 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:08:19.535 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T13:08:19.536 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:08:19.536 INFO:teuthology.orchestra.run.smithi096.stdout:Install 13 Packages 2023-09-22T13:08:19.536 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:08:19.537 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 1.9 M 2023-09-22T13:08:19.537 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 24 M 2023-09-22T13:08:19.537 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T13:08:20.050 INFO:teuthology.orchestra.run.smithi096.stdout:(1/13): ceph-mgr-rook-16.2.14-5.g31c126b7.el8.n 125 kB/s | 27 kB 00:00 2023-09-22T13:08:20.558 INFO:teuthology.orchestra.run.smithi096.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 36 kB/s | 27 kB 00:00 2023-09-22T13:08:21.208 INFO:teuthology.orchestra.run.smithi096.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 14 kB/s | 20 kB 00:01 2023-09-22T13:08:23.092 INFO:teuthology.orchestra.run.smithi096.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 82 kB/s | 155 kB 00:01 2023-09-22T13:08:23.759 INFO:teuthology.orchestra.run.smithi096.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 64 kB/s | 43 kB 00:00 2023-09-22T13:08:23.784 INFO:teuthology.orchestra.run.smithi096.stdout:(6/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 34 kB/s | 126 kB 00:03 2023-09-22T13:08:24.142 INFO:teuthology.orchestra.run.smithi096.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 31 kB/s | 110 kB 00:03 2023-09-22T13:08:24.192 INFO:teuthology.orchestra.run.smithi096.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 40 kB/s | 16 kB 00:00 2023-09-22T13:08:24.218 INFO:teuthology.orchestra.run.smithi096.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 72 kB/s | 33 kB 00:00 2023-09-22T13:08:24.435 INFO:teuthology.orchestra.run.smithi096.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 296 kB/s | 64 kB 00:00 2023-09-22T13:08:24.585 INFO:teuthology.orchestra.run.smithi096.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 408 kB/s | 61 kB 00:00 2023-09-22T13:08:24.719 INFO:teuthology.orchestra.run.smithi096.stdout:(12/13): python3-google-auth-1.1.1-10.el8.noarc 141 kB/s | 81 kB 00:00 2023-09-22T13:08:24.886 INFO:teuthology.orchestra.run.smithi096.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.7 MB/s | 1.2 MB 00:00 2023-09-22T13:08:24.889 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:08:24.889 INFO:teuthology.orchestra.run.smithi096.stdout:Total 365 kB/s | 1.9 MB 00:05 2023-09-22T13:08:24.948 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T13:08:24.964 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T13:08:24.964 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T13:08:25.051 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T13:08:25.054 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T13:08:25.542 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T13:08:25.764 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-22T13:08:25.953 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-22T13:08:26.139 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-22T13:08:26.343 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-22T13:08:26.542 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-22T13:08:26.773 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-22T13:08:27.064 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-22T13:08:27.307 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-22T13:08:28.343 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-22T13:08:28.600 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-22T13:08:28.818 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-22T13:08:29.001 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-22T13:08:29.160 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-rook-2:16.2.14-5.g31c126b7.el8.noarch 13/13 2023-09-22T13:08:29.456 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-5.g31c126b7.el8.noarch 13/13 2023-09-22T13:08:29.456 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-rook-2:16.2.14-5.g31c126b7.el8.noarch 1/13 2023-09-22T13:08:29.456 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-22T13:08:29.456 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-22T13:08:29.456 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-22T13:08:29.456 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-22T13:08:29.457 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-22T13:08:29.457 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-22T13:08:29.457 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-22T13:08:29.457 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-22T13:08:29.457 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-22T13:08:29.457 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-22T13:08:29.457 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-22T13:08:30.134 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-22T13:08:30.134 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:08:30.134 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T13:08:30.134 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook-2:16.2.14-5.g31c126b7.el8.noarch 2023-09-22T13:08:30.134 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-22T13:08:30.134 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-22T13:08:30.134 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-22T13:08:30.135 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-22T13:08:30.135 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-22T13:08:30.135 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-22T13:08:30.135 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-22T13:08:30.135 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-22T13:08:30.135 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-22T13:08:30.135 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-22T13:08:30.135 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-22T13:08:30.135 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-22T13:08:30.135 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:08:30.135 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:08:30.249 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-cephadm 2023-09-22T13:08:31.099 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:18:14 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T13:08:32.662 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T13:08:32.663 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:08:32.663 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2023-09-22T13:08:32.663 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:08:32.663 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T13:08:32.663 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm noarch 2:16.2.14-5.g31c126b7.el8 ceph-noarch 116 k 2023-09-22T13:08:32.663 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2023-09-22T13:08:32.664 INFO:teuthology.orchestra.run.smithi096.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-09-22T13:08:32.664 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-22T13:08:32.664 INFO:teuthology.orchestra.run.smithi096.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-09-22T13:08:32.664 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-22T13:08:32.664 INFO:teuthology.orchestra.run.smithi096.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-09-22T13:08:32.664 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:08:32.664 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T13:08:32.664 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:08:32.664 INFO:teuthology.orchestra.run.smithi096.stdout:Install 6 Packages 2023-09-22T13:08:32.664 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:08:32.665 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 5.8 M 2023-09-22T13:08:32.665 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 24 M 2023-09-22T13:08:32.665 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T13:08:33.185 INFO:teuthology.orchestra.run.smithi096.stdout:(1/6): ceph-mgr-cephadm-16.2.14-5.g31c126b7.el8 397 kB/s | 116 kB 00:00 2023-09-22T13:08:33.386 INFO:teuthology.orchestra.run.smithi096.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 106 kB/s | 21 kB 00:00 2023-09-22T13:08:33.561 INFO:teuthology.orchestra.run.smithi096.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.0 MB/s | 355 kB 00:00 2023-09-22T13:08:33.628 INFO:teuthology.orchestra.run.smithi096.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 732 kB/s | 49 kB 00:00 2023-09-22T13:08:44.749 INFO:teuthology.orchestra.run.smithi105.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 21 kB/s | 14 MB 11:14 2023-09-22T13:08:44.749 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:08:44.749 INFO:teuthology.orchestra.run.smithi105.stdout:Total 43 kB/s | 37 MB 14:32 2023-09-22T13:08:44.976 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-09-22T13:08:44.994 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-09-22T13:08:44.995 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-09-22T13:08:45.330 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-09-22T13:08:45.334 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-09-22T13:08:45.896 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-09-22T13:08:46.038 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-22T13:08:46.175 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-22T13:08:46.313 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-22T13:08:46.914 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-22T13:08:47.105 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-22T13:08:47.710 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-22T13:08:47.846 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-22T13:08:48.550 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-22T13:08:48.883 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-22T13:08:49.138 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-22T13:08:50.903 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-22T13:08:52.198 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-22T13:08:52.679 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-5.g31c126b7. 8/8 2023-09-22T13:08:53.547 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-5.g31c126b7. 8/8 2023-09-22T13:08:53.547 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-5.g31c126b7. 1/8 2023-09-22T13:08:53.547 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-22T13:08:53.547 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-22T13:08:53.548 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-22T13:08:53.548 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-22T13:08:53.548 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-22T13:08:53.548 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-22T13:08:53.658 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-22T13:08:53.658 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:08:53.658 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-09-22T13:08:53.658 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local-2:16.2.14-5.g31c126b7.el8.noarch 2023-09-22T13:08:53.658 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-22T13:08:53.658 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-22T13:08:53.659 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-22T13:08:53.659 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-22T13:08:53.659 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-22T13:08:53.659 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-22T13:08:53.659 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-22T13:08:53.659 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:08:53.659 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T13:08:53.785 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-rook 2023-09-22T13:08:55.011 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:17:27 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T13:08:56.579 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T13:08:56.580 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:08:56.580 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2023-09-22T13:08:56.580 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:08:56.580 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-09-22T13:08:56.580 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook noarch 2:16.2.14-5.g31c126b7.el8 ceph-noarch 27 k 2023-09-22T13:08:56.581 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2023-09-22T13:08:56.581 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-22T13:08:56.581 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-22T13:08:56.581 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-22T13:08:56.581 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-22T13:08:56.581 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-22T13:08:56.581 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-22T13:08:56.581 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-22T13:08:56.581 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-22T13:08:56.581 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-22T13:08:56.581 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-22T13:08:56.581 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-22T13:08:56.582 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-22T13:08:56.582 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:08:56.582 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-09-22T13:08:56.582 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:08:56.582 INFO:teuthology.orchestra.run.smithi105.stdout:Install 13 Packages 2023-09-22T13:08:56.582 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:08:56.583 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 1.9 M 2023-09-22T13:08:56.583 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 24 M 2023-09-22T13:08:56.583 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-09-22T13:08:57.212 INFO:teuthology.orchestra.run.smithi105.stdout:(1/13): ceph-mgr-rook-16.2.14-5.g31c126b7.el8.n 126 kB/s | 27 kB 00:00 2023-09-22T13:08:57.750 INFO:teuthology.orchestra.run.smithi105.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 27 kB/s | 20 kB 00:00 2023-09-22T13:08:59.079 INFO:teuthology.orchestra.run.smithi105.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 13 kB/s | 27 kB 00:02 2023-09-22T13:09:01.238 INFO:teuthology.orchestra.run.smithi105.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 72 kB/s | 155 kB 00:02 2023-09-22T13:09:02.079 INFO:teuthology.orchestra.run.smithi105.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 26 kB/s | 126 kB 00:04 2023-09-22T13:09:02.321 INFO:teuthology.orchestra.run.smithi105.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 137 kB/s | 33 kB 00:00 2023-09-22T13:09:02.405 INFO:teuthology.orchestra.run.smithi105.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 195 kB/s | 16 kB 00:00 2023-09-22T13:09:02.547 INFO:teuthology.orchestra.run.smithi105.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 574 kB/s | 81 kB 00:00 2023-09-22T13:09:02.822 INFO:teuthology.orchestra.run.smithi105.stdout:(9/13): python3-kubernetes-11.0.0-6.el8.noarch. 4.2 MB/s | 1.2 MB 00:00 2023-09-22T13:09:02.897 INFO:teuthology.orchestra.run.smithi105.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 859 kB/s | 64 kB 00:00 2023-09-22T13:09:02.972 INFO:teuthology.orchestra.run.smithi105.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 815 kB/s | 61 kB 00:00 2023-09-22T13:09:07.222 INFO:teuthology.orchestra.run.smithi105.stdout:(12/13): python3-pyasn1-modules-0.3.7-6.el8.noa 12 kB/s | 110 kB 00:09 2023-09-22T13:09:07.246 INFO:teuthology.orchestra.run.smithi096.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 16 kB/s | 538 kB 00:34 2023-09-22T13:09:34.315 INFO:teuthology.orchestra.run.smithi105.stdout:(13/13): python3-requests-oauthlib-1.0.0-1.el8. 1.3 kB/s | 43 kB 00:33 2023-09-22T13:09:34.317 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:09:34.318 INFO:teuthology.orchestra.run.smithi105.stdout:Total 52 kB/s | 1.9 MB 00:37 2023-09-22T13:09:34.374 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-09-22T13:09:34.390 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-09-22T13:09:34.390 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-09-22T13:09:34.476 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-09-22T13:09:34.480 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-09-22T13:09:34.957 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-09-22T13:09:35.217 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-22T13:09:35.415 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-22T13:09:35.642 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-22T13:09:35.880 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-22T13:09:36.087 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-22T13:09:36.326 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-22T13:09:36.600 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-22T13:09:36.844 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-22T13:09:37.883 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-22T13:09:38.155 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-22T13:09:38.355 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-22T13:09:38.547 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-22T13:09:38.706 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-rook-2:16.2.14-5.g31c126b7.el8.noarch 13/13 2023-09-22T13:09:38.997 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-5.g31c126b7.el8.noarch 13/13 2023-09-22T13:09:38.997 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-rook-2:16.2.14-5.g31c126b7.el8.noarch 1/13 2023-09-22T13:09:38.997 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-22T13:09:38.998 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-22T13:09:38.998 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-22T13:09:38.998 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-22T13:09:38.998 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-22T13:09:38.998 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-22T13:09:38.998 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-22T13:09:38.998 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-22T13:09:38.998 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-22T13:09:38.999 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-22T13:09:38.999 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-22T13:09:39.667 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-22T13:09:39.667 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:09:39.667 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-09-22T13:09:39.667 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook-2:16.2.14-5.g31c126b7.el8.noarch 2023-09-22T13:09:39.667 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-22T13:09:39.668 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-22T13:09:39.668 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-22T13:09:39.668 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-22T13:09:39.668 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-22T13:09:39.668 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-22T13:09:39.668 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-22T13:09:39.668 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-22T13:09:39.669 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-22T13:09:39.669 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-22T13:09:39.669 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-22T13:09:39.669 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-22T13:09:39.669 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:09:39.669 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T13:09:39.798 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-cephadm 2023-09-22T13:09:40.650 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:18:12 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T13:09:42.234 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T13:09:42.235 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:09:42.235 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2023-09-22T13:09:42.235 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:09:42.235 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-09-22T13:09:42.236 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm noarch 2:16.2.14-5.g31c126b7.el8 ceph-noarch 116 k 2023-09-22T13:09:42.236 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2023-09-22T13:09:42.236 INFO:teuthology.orchestra.run.smithi105.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-09-22T13:09:42.236 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-22T13:09:42.236 INFO:teuthology.orchestra.run.smithi105.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-09-22T13:09:42.236 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-22T13:09:42.236 INFO:teuthology.orchestra.run.smithi105.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-09-22T13:09:42.237 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:09:42.237 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-09-22T13:09:42.237 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:09:42.237 INFO:teuthology.orchestra.run.smithi105.stdout:Install 6 Packages 2023-09-22T13:09:42.237 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:09:42.238 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 5.8 M 2023-09-22T13:09:42.238 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 24 M 2023-09-22T13:09:42.238 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-09-22T13:09:43.006 INFO:teuthology.orchestra.run.smithi105.stdout:(1/6): ceph-mgr-cephadm-16.2.14-5.g31c126b7.el8 301 kB/s | 116 kB 00:00 2023-09-22T13:09:43.090 INFO:teuthology.orchestra.run.smithi105.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 254 kB/s | 21 kB 00:00 2023-09-22T13:09:43.190 INFO:teuthology.orchestra.run.smithi105.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 3.5 MB/s | 355 kB 00:00 2023-09-22T13:09:43.241 INFO:teuthology.orchestra.run.smithi105.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 978 kB/s | 49 kB 00:00 2023-09-22T13:10:18.074 INFO:teuthology.orchestra.run.smithi105.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 15 kB/s | 538 kB 00:35 2023-09-22T13:12:47.892 INFO:teuthology.orchestra.run.smithi096.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 19 kB/s | 4.8 MB 04:14 2023-09-22T13:12:47.893 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:12:47.893 INFO:teuthology.orchestra.run.smithi096.stdout:Total 23 kB/s | 5.8 MB 04:15 2023-09-22T13:12:47.945 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T13:12:47.955 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T13:12:47.956 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T13:12:48.068 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T13:12:48.074 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T13:12:48.405 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T13:12:48.677 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-09-22T13:12:48.939 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-09-22T13:12:49.870 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-09-22T13:12:50.214 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-09-22T13:12:50.460 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-09-22T13:12:50.636 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-5.g31c126b7.el8.noarch 6/6 2023-09-22T13:12:50.921 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-5.g31c126b7.el8.noarch 6/6 2023-09-22T13:12:50.921 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-5.g31c126b7.el8.noarch 1/6 2023-09-22T13:12:50.921 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-09-22T13:12:50.921 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-09-22T13:12:50.921 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-09-22T13:12:50.921 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-09-22T13:12:51.534 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-09-22T13:12:51.534 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:12:51.534 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T13:12:51.534 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm-2:16.2.14-5.g31c126b7.el8.noarch 2023-09-22T13:12:51.534 INFO:teuthology.orchestra.run.smithi096.stdout: python3-apipkg-1.5-6.el8.noarch 2023-09-22T13:12:51.535 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-22T13:12:51.535 INFO:teuthology.orchestra.run.smithi096.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-09-22T13:12:51.535 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-22T13:12:51.535 INFO:teuthology.orchestra.run.smithi096.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-09-22T13:12:51.535 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:12:51.535 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:12:51.633 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-fuse 2023-09-22T13:12:52.489 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:22:35 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T13:12:54.058 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T13:12:54.059 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:12:54.059 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2023-09-22T13:12:54.059 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:12:54.059 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T13:12:54.059 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse x86_64 2:16.2.14-5.g31c126b7.el8 ceph 910 k 2023-09-22T13:12:54.059 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2023-09-22T13:12:54.060 INFO:teuthology.orchestra.run.smithi096.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-22T13:12:54.060 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:12:54.060 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T13:12:54.060 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:12:54.060 INFO:teuthology.orchestra.run.smithi096.stdout:Install 2 Packages 2023-09-22T13:12:54.060 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:12:54.061 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 993 k 2023-09-22T13:12:54.061 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 2.8 M 2023-09-22T13:12:54.061 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T13:12:54.375 INFO:teuthology.orchestra.run.smithi096.stdout:(1/2): ceph-fuse-16.2.14-5.g31c126b7.el8.x86_64 3.2 MB/s | 910 kB 00:00 2023-09-22T13:13:11.903 INFO:teuthology.orchestra.run.smithi105.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 23 kB/s | 4.8 MB 03:29 2023-09-22T13:13:11.904 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:13:11.904 INFO:teuthology.orchestra.run.smithi105.stdout:Total 29 kB/s | 5.8 MB 03:29 2023-09-22T13:13:11.956 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-09-22T13:13:11.966 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-09-22T13:13:11.966 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-09-22T13:13:12.067 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-09-22T13:13:12.070 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-09-22T13:13:12.399 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-09-22T13:13:12.660 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-09-22T13:13:12.905 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-09-22T13:13:13.823 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-09-22T13:13:14.173 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-09-22T13:13:14.452 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-09-22T13:13:14.627 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-5.g31c126b7.el8.noarch 6/6 2023-09-22T13:13:14.898 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-5.g31c126b7.el8.noarch 6/6 2023-09-22T13:13:14.898 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-5.g31c126b7.el8.noarch 1/6 2023-09-22T13:13:14.898 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-09-22T13:13:14.898 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-09-22T13:13:14.898 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-09-22T13:13:14.898 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-09-22T13:13:15.495 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-09-22T13:13:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:13:15.496 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-09-22T13:13:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm-2:16.2.14-5.g31c126b7.el8.noarch 2023-09-22T13:13:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: python3-apipkg-1.5-6.el8.noarch 2023-09-22T13:13:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-22T13:13:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-09-22T13:13:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-22T13:13:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-09-22T13:13:15.496 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:13:15.496 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T13:13:15.599 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-fuse 2023-09-22T13:13:16.454 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:21:48 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T13:13:18.023 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T13:13:18.023 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:13:18.023 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2023-09-22T13:13:18.024 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:13:18.024 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-09-22T13:13:18.024 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse x86_64 2:16.2.14-5.g31c126b7.el8 ceph 910 k 2023-09-22T13:13:18.024 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2023-09-22T13:13:18.024 INFO:teuthology.orchestra.run.smithi105.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-22T13:13:18.024 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:13:18.024 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-09-22T13:13:18.024 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:13:18.024 INFO:teuthology.orchestra.run.smithi105.stdout:Install 2 Packages 2023-09-22T13:13:18.024 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:13:18.025 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 993 k 2023-09-22T13:13:18.025 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 2.8 M 2023-09-22T13:13:18.025 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-09-22T13:13:18.470 INFO:teuthology.orchestra.run.smithi105.stdout:(1/2): ceph-fuse-16.2.14-5.g31c126b7.el8.x86_64 3.5 MB/s | 910 kB 00:00 2023-09-22T13:13:52.535 INFO:teuthology.orchestra.run.smithi105.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 2.4 kB/s | 83 kB 00:34 2023-09-22T13:13:52.535 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:13:52.536 INFO:teuthology.orchestra.run.smithi105.stdout:Total 29 kB/s | 993 kB 00:34 2023-09-22T13:13:52.547 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-09-22T13:13:52.569 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-09-22T13:13:52.569 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-09-22T13:13:52.662 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-09-22T13:13:52.665 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-09-22T13:13:52.834 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-09-22T13:13:53.105 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-22T13:13:53.272 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-fuse-2:16.2.14-5.g31c126b7.el8.x86_64 2/2 2023-09-22T13:13:53.660 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-fuse-2:16.2.14-5.g31c126b7.el8.x86_64 2/2 2023-09-22T13:13:53.660 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-fuse-2:16.2.14-5.g31c126b7.el8.x86_64 1/2 2023-09-22T13:13:54.257 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-22T13:13:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:13:54.258 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-09-22T13:13:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse-2:16.2.14-5.g31c126b7.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-22T13:13:54.258 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:13:54.258 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T13:13:54.452 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librados-devel 2023-09-22T13:13:55.308 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:22:27 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T13:13:56.867 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T13:13:56.868 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:13:56.868 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2023-09-22T13:13:56.868 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:13:56.868 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-09-22T13:13:56.868 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel x86_64 2:16.2.14-5.g31c126b7.el8 ceph 131 k 2023-09-22T13:13:56.868 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:13:56.868 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-09-22T13:13:56.868 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:13:56.868 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2023-09-22T13:13:56.868 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:13:56.869 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 131 k 2023-09-22T13:13:56.869 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 432 k 2023-09-22T13:13:56.869 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-09-22T13:13:57.056 INFO:teuthology.orchestra.run.smithi105.stdout:librados-devel-16.2.14-5.g31c126b7.el8.x86_64.r 702 kB/s | 131 kB 00:00 2023-09-22T13:13:57.056 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:13:57.056 INFO:teuthology.orchestra.run.smithi105.stdout:Total 698 kB/s | 131 kB 00:00 2023-09-22T13:13:57.057 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-09-22T13:13:57.062 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-09-22T13:13:57.062 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-09-22T13:13:57.112 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-09-22T13:13:57.115 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-09-22T13:13:57.223 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-09-22T13:13:57.438 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librados-devel-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:13:57.712 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados-devel-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:13:58.304 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados-devel-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:13:58.304 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:13:58.304 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-09-22T13:13:58.305 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T13:13:58.305 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:13:58.305 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T13:13:58.508 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install libcephfs2 2023-09-22T13:13:59.367 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:22:31 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T13:14:00.762 INFO:teuthology.orchestra.run.smithi105.stdout:Package libcephfs2-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T13:14:00.942 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T13:14:00.943 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2023-09-22T13:14:00.943 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T13:14:01.032 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install libcephfs-devel 2023-09-22T13:14:01.881 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:22:33 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T13:14:03.448 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T13:14:03.448 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:14:03.448 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2023-09-22T13:14:03.449 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:14:03.449 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-09-22T13:14:03.449 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel x86_64 2:16.2.14-5.g31c126b7.el8 ceph 25 k 2023-09-22T13:14:03.449 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:14:03.449 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-09-22T13:14:03.449 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:14:03.449 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2023-09-22T13:14:03.449 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:14:03.450 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 25 k 2023-09-22T13:14:03.450 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 107 k 2023-09-22T13:14:03.450 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-09-22T13:14:03.675 INFO:teuthology.orchestra.run.smithi105.stdout:libcephfs-devel-16.2.14-5.g31c126b7.el8.x86_64. 113 kB/s | 25 kB 00:00 2023-09-22T13:14:03.676 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:14:03.676 INFO:teuthology.orchestra.run.smithi105.stdout:Total 113 kB/s | 25 kB 00:00 2023-09-22T13:14:03.676 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-09-22T13:14:03.680 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-09-22T13:14:03.680 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-09-22T13:14:03.682 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-09-22T13:14:03.685 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-09-22T13:14:03.739 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-09-22T13:14:03.928 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs-devel-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:14:04.180 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:14:04.754 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs-devel-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:14:04.754 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:14:04.754 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-09-22T13:14:04.754 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T13:14:04.754 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:14:04.754 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T13:14:04.861 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librados2 2023-09-22T13:14:05.726 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:22:37 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T13:14:07.120 INFO:teuthology.orchestra.run.smithi105.stdout:Package librados2-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T13:14:07.298 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T13:14:07.300 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2023-09-22T13:14:07.300 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T13:14:07.389 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librbd1 2023-09-22T13:14:08.240 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:22:40 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T13:14:09.638 INFO:teuthology.orchestra.run.smithi105.stdout:Package librbd1-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T13:14:09.819 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T13:14:09.820 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2023-09-22T13:14:09.820 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T13:14:09.905 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rados 2023-09-22T13:14:10.754 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:22:42 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T13:14:12.148 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rados-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T13:14:12.329 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T13:14:12.330 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2023-09-22T13:14:12.330 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T13:14:12.427 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rgw 2023-09-22T13:14:13.289 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:22:45 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T13:14:14.653 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rgw-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T13:14:14.834 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T13:14:14.835 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2023-09-22T13:14:14.835 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T13:14:14.935 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-cephfs 2023-09-22T13:14:15.784 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:22:47 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T13:14:17.181 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-cephfs-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T13:14:17.365 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T13:14:17.366 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2023-09-22T13:14:17.366 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T13:14:17.460 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rbd 2023-09-22T13:14:18.314 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:22:50 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T13:14:19.700 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rbd-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T13:14:19.880 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T13:14:19.881 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2023-09-22T13:14:19.881 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T13:14:19.976 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-fuse 2023-09-22T13:14:20.832 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:22:52 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T13:14:22.381 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T13:14:22.382 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:14:22.382 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2023-09-22T13:14:22.383 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:14:22.383 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-09-22T13:14:22.383 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse x86_64 2:16.2.14-5.g31c126b7.el8 ceph 88 k 2023-09-22T13:14:22.383 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:14:22.383 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-09-22T13:14:22.383 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:14:22.383 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2023-09-22T13:14:22.383 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:14:22.383 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 88 k 2023-09-22T13:14:22.383 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 240 k 2023-09-22T13:14:22.383 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-09-22T13:14:22.686 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-fuse-16.2.14-5.g31c126b7.el8.x86_64.rpm 292 kB/s | 88 kB 00:00 2023-09-22T13:14:22.686 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:14:22.686 INFO:teuthology.orchestra.run.smithi105.stdout:Total 291 kB/s | 88 kB 00:00 2023-09-22T13:14:22.687 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-09-22T13:14:22.696 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-09-22T13:14:22.696 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-09-22T13:14:22.745 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-09-22T13:14:22.748 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-09-22T13:14:22.854 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-09-22T13:14:23.026 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-fuse-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:14:23.293 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-fuse-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:14:23.908 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-fuse-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:14:23.908 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:14:23.908 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-09-22T13:14:23.908 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T13:14:23.908 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:14:23.908 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T13:14:24.086 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-mirror 2023-09-22T13:14:24.938 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:22:56 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T13:14:26.511 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T13:14:26.512 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:14:26.512 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2023-09-22T13:14:26.512 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:14:26.512 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-09-22T13:14:26.512 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror x86_64 2:16.2.14-5.g31c126b7.el8 ceph 3.7 M 2023-09-22T13:14:26.512 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:14:26.512 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-09-22T13:14:26.512 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:14:26.512 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2023-09-22T13:14:26.512 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:14:26.513 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 3.7 M 2023-09-22T13:14:26.513 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 15 M 2023-09-22T13:14:26.513 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-09-22T13:14:27.271 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-mirror-16.2.14-5.g31c126b7.el8.x86_64.rpm 4.8 MB/s | 3.7 MB 00:00 2023-09-22T13:14:27.272 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:14:27.272 INFO:teuthology.orchestra.run.smithi105.stdout:Total 4.8 MB/s | 3.7 MB 00:00 2023-09-22T13:14:27.272 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-09-22T13:14:27.284 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-09-22T13:14:27.284 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-09-22T13:14:27.349 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-09-22T13:14:27.352 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-09-22T13:14:27.801 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-09-22T13:14:28.001 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-mirror-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:14:28.420 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-mirror-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:14:29.017 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-mirror-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:14:29.017 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:14:29.017 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-09-22T13:14:29.017 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T13:14:29.018 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:14:29.018 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T13:14:29.213 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-nbd 2023-09-22T13:14:30.068 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:23:02 ago on Fri 22 Sep 2023 12:51:28 PM UTC. 2023-09-22T13:14:31.649 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-09-22T13:14:31.650 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:14:31.651 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2023-09-22T13:14:31.651 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:14:31.651 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-09-22T13:14:31.651 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd x86_64 2:16.2.14-5.g31c126b7.el8 ceph 173 k 2023-09-22T13:14:31.651 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:14:31.651 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-09-22T13:14:31.651 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-09-22T13:14:31.651 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2023-09-22T13:14:31.651 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:14:31.651 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 173 k 2023-09-22T13:14:31.651 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 515 k 2023-09-22T13:14:31.651 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-09-22T13:14:31.941 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-nbd-16.2.14-5.g31c126b7.el8.x86_64.rpm 599 kB/s | 173 kB 00:00 2023-09-22T13:14:31.941 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:14:31.941 INFO:teuthology.orchestra.run.smithi105.stdout:Total 596 kB/s | 173 kB 00:00 2023-09-22T13:14:31.942 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-09-22T13:14:31.952 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-09-22T13:14:31.952 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-09-22T13:14:32.002 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-09-22T13:14:32.005 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-09-22T13:14:32.111 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-09-22T13:14:32.291 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-nbd-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:14:32.573 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-nbd-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:14:33.160 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-nbd-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:14:33.160 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:14:33.160 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-09-22T13:14:33.160 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T13:14:33.161 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:14:33.161 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-09-22T13:14:33.376 DEBUG:teuthology.parallel:result is None 2023-09-22T13:15:02.969 INFO:teuthology.orchestra.run.smithi096.stdout:[MIRROR] fuse-2.9.7-17.el8.x86_64.rpm: Curl error (7): Couldn't connect to server for http://mirrors.unifiedlayer.com/centos/8-stream/BaseOS/x86_64/os/Packages/fuse-2.9.7-17.el8.x86_64.rpm [Failed to connect to mirrors.unifiedlayer.com port 80: Connection timed out] 2023-09-22T13:15:03.150 INFO:teuthology.orchestra.run.smithi096.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 658 B/s | 83 kB 02:09 2023-09-22T13:15:03.150 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:15:03.150 INFO:teuthology.orchestra.run.smithi096.stdout:Total 7.7 kB/s | 993 kB 02:09 2023-09-22T13:15:03.161 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T13:15:03.183 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T13:15:03.191 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T13:15:03.273 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T13:15:03.276 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T13:15:03.441 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T13:15:03.689 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-22T13:15:03.898 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-fuse-2:16.2.14-5.g31c126b7.el8.x86_64 2/2 2023-09-22T13:15:04.280 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-fuse-2:16.2.14-5.g31c126b7.el8.x86_64 2/2 2023-09-22T13:15:04.281 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-fuse-2:16.2.14-5.g31c126b7.el8.x86_64 1/2 2023-09-22T13:15:04.922 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-22T13:15:04.922 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:04.922 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T13:15:04.922 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse-2:16.2.14-5.g31c126b7.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-22T13:15:04.922 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:04.922 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:15:05.117 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librados-devel 2023-09-22T13:15:05.976 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:24:48 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T13:15:07.541 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T13:15:07.542 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:15:07.542 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2023-09-22T13:15:07.542 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:15:07.542 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T13:15:07.542 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel x86_64 2:16.2.14-5.g31c126b7.el8 ceph 131 k 2023-09-22T13:15:07.542 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:07.542 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T13:15:07.542 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:15:07.542 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2023-09-22T13:15:07.543 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:07.543 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 131 k 2023-09-22T13:15:07.543 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 432 k 2023-09-22T13:15:07.543 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T13:15:07.845 INFO:teuthology.orchestra.run.smithi096.stdout:librados-devel-16.2.14-5.g31c126b7.el8.x86_64.r 434 kB/s | 131 kB 00:00 2023-09-22T13:15:07.846 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:15:07.846 INFO:teuthology.orchestra.run.smithi096.stdout:Total 432 kB/s | 131 kB 00:00 2023-09-22T13:15:07.846 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T13:15:07.852 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T13:15:07.852 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T13:15:07.902 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T13:15:07.905 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T13:15:08.011 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T13:15:08.246 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librados-devel-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:15:08.534 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados-devel-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:15:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados-devel-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:15:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:09.118 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T13:15:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T13:15:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:09.119 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:15:09.337 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install libcephfs2 2023-09-22T13:15:10.200 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:24:53 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T13:15:11.606 INFO:teuthology.orchestra.run.smithi096.stdout:Package libcephfs2-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T13:15:11.789 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T13:15:11.790 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2023-09-22T13:15:11.790 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:15:11.890 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install libcephfs-devel 2023-09-22T13:15:12.739 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:24:55 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T13:15:14.322 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T13:15:14.323 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:15:14.323 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2023-09-22T13:15:14.323 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:15:14.323 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T13:15:14.323 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel x86_64 2:16.2.14-5.g31c126b7.el8 ceph 25 k 2023-09-22T13:15:14.323 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:14.323 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T13:15:14.323 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:15:14.324 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2023-09-22T13:15:14.324 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:14.324 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 25 k 2023-09-22T13:15:14.324 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 107 k 2023-09-22T13:15:14.324 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T13:15:14.472 INFO:teuthology.orchestra.run.smithi096.stdout:libcephfs-devel-16.2.14-5.g31c126b7.el8.x86_64. 172 kB/s | 25 kB 00:00 2023-09-22T13:15:14.473 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:15:14.473 INFO:teuthology.orchestra.run.smithi096.stdout:Total 170 kB/s | 25 kB 00:00 2023-09-22T13:15:14.473 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T13:15:14.477 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T13:15:14.477 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T13:15:14.479 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T13:15:14.482 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T13:15:14.528 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T13:15:14.719 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs-devel-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:15:14.986 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:15:15.539 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs-devel-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:15:15.540 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:15.540 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T13:15:15.540 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T13:15:15.540 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:15.540 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:15:15.652 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librados2 2023-09-22T13:15:16.507 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:24:59 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T13:15:17.877 INFO:teuthology.orchestra.run.smithi096.stdout:Package librados2-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T13:15:18.058 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T13:15:18.060 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2023-09-22T13:15:18.060 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:15:18.157 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librbd1 2023-09-22T13:15:19.005 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:25:02 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T13:15:20.395 INFO:teuthology.orchestra.run.smithi096.stdout:Package librbd1-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T13:15:20.575 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T13:15:20.576 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2023-09-22T13:15:20.576 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:15:20.682 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rados 2023-09-22T13:15:21.537 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:25:04 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T13:15:22.930 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rados-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T13:15:23.112 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T13:15:23.113 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2023-09-22T13:15:23.113 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:15:23.215 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rgw 2023-09-22T13:15:24.065 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:25:07 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T13:15:25.446 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rgw-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T13:15:25.626 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T13:15:25.627 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2023-09-22T13:15:25.627 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:15:25.722 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-cephfs 2023-09-22T13:15:26.572 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:25:09 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T13:15:27.963 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-cephfs-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T13:15:28.144 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T13:15:28.145 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2023-09-22T13:15:28.145 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:15:28.248 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rbd 2023-09-22T13:15:29.094 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:25:12 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T13:15:30.505 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rbd-2:16.2.14-5.g31c126b7.el8.x86_64 is already installed. 2023-09-22T13:15:30.685 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T13:15:30.686 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2023-09-22T13:15:30.687 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:15:30.784 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-fuse 2023-09-22T13:15:31.633 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:25:14 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T13:15:33.204 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T13:15:33.205 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:15:33.205 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2023-09-22T13:15:33.205 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:15:33.205 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T13:15:33.205 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse x86_64 2:16.2.14-5.g31c126b7.el8 ceph 88 k 2023-09-22T13:15:33.205 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:33.206 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T13:15:33.206 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:15:33.206 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2023-09-22T13:15:33.206 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:33.206 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 88 k 2023-09-22T13:15:33.207 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 240 k 2023-09-22T13:15:33.207 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T13:15:33.381 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-fuse-16.2.14-5.g31c126b7.el8.x86_64.rpm 506 kB/s | 88 kB 00:00 2023-09-22T13:15:33.382 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:15:33.382 INFO:teuthology.orchestra.run.smithi096.stdout:Total 503 kB/s | 88 kB 00:00 2023-09-22T13:15:33.382 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T13:15:33.391 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T13:15:33.392 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T13:15:33.440 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T13:15:33.443 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T13:15:33.542 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T13:15:33.724 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-fuse-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:15:34.005 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-fuse-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:15:34.623 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-fuse-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:15:34.623 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:34.623 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T13:15:34.623 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T13:15:34.623 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:34.623 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:15:34.812 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-mirror 2023-09-22T13:15:35.664 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:25:18 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T13:15:37.244 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T13:15:37.245 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:15:37.245 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2023-09-22T13:15:37.245 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:15:37.245 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T13:15:37.245 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror x86_64 2:16.2.14-5.g31c126b7.el8 ceph 3.7 M 2023-09-22T13:15:37.245 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:37.245 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T13:15:37.245 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:15:37.245 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2023-09-22T13:15:37.245 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:37.246 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 3.7 M 2023-09-22T13:15:37.246 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 15 M 2023-09-22T13:15:37.246 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T13:15:37.801 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-mirror-16.2.14-5.g31c126b7.el8.x86_64.rpm 6.6 MB/s | 3.7 MB 00:00 2023-09-22T13:15:37.802 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:15:37.802 INFO:teuthology.orchestra.run.smithi096.stdout:Total 6.6 MB/s | 3.7 MB 00:00 2023-09-22T13:15:37.803 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T13:15:37.814 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T13:15:37.814 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T13:15:37.878 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T13:15:37.881 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T13:15:38.324 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T13:15:38.532 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-mirror-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:15:38.945 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-mirror-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:15:39.539 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-mirror-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:15:39.539 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:39.540 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T13:15:39.540 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T13:15:39.540 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:39.540 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:15:39.738 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-nbd 2023-09-22T13:15:40.595 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:25:23 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T13:15:42.166 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T13:15:42.166 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:15:42.167 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2023-09-22T13:15:42.167 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:15:42.167 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T13:15:42.167 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd x86_64 2:16.2.14-5.g31c126b7.el8 ceph 173 k 2023-09-22T13:15:42.167 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:42.167 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T13:15:42.167 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:15:42.167 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2023-09-22T13:15:42.167 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:42.168 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 173 k 2023-09-22T13:15:42.168 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 515 k 2023-09-22T13:15:42.168 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T13:15:42.356 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-nbd-16.2.14-5.g31c126b7.el8.x86_64.rpm 920 kB/s | 173 kB 00:00 2023-09-22T13:15:42.357 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:15:42.357 INFO:teuthology.orchestra.run.smithi096.stdout:Total 914 kB/s | 173 kB 00:00 2023-09-22T13:15:42.357 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T13:15:42.368 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T13:15:42.368 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T13:15:42.416 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T13:15:42.420 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T13:15:42.532 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T13:15:42.696 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-nbd-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:15:42.970 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-nbd-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:15:43.562 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-nbd-2:16.2.14-5.g31c126b7.el8.x86_64 1/1 2023-09-22T13:15:43.563 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:43.563 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T13:15:43.563 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd-2:16.2.14-5.g31c126b7.el8.x86_64 2023-09-22T13:15:43.563 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:43.563 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:15:43.740 DEBUG:teuthology.parallel:result is None 2023-09-22T13:15:43.741 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=31c126b7c1c0c06b66d9d19c5bfc51c46b503392 2023-09-22T13:15:43.885 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-22T13:15:43.908 INFO:teuthology.orchestra.run.smithi096.stdout:16.2.14-5.g31c126b7.el8 2023-09-22T13:15:43.908 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-5.g31c126b7.el8 2023-09-22T13:15:43.909 INFO:teuthology.task.install:The correct ceph version 16.2.14-5.g31c126b7 is installed. 2023-09-22T13:15:43.909 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=31c126b7c1c0c06b66d9d19c5bfc51c46b503392 2023-09-22T13:15:44.051 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-22T13:15:44.078 INFO:teuthology.orchestra.run.smithi105.stdout:16.2.14-5.g31c126b7.el8 2023-09-22T13:15:44.078 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-5.g31c126b7.el8 2023-09-22T13:15:44.078 INFO:teuthology.task.install:The correct ceph version 16.2.14-5.g31c126b7 is installed. 2023-09-22T13:15:44.079 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-22T13:15:44.079 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:15:44.079 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-22T13:15:44.107 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:15:44.107 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-22T13:15:44.151 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-22T13:15:44.151 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:15:44.151 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/daemon-helper 2023-09-22T13:15:44.178 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-22T13:15:44.246 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:15:44.246 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/daemon-helper 2023-09-22T13:15:44.276 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-22T13:15:44.345 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-22T13:15:44.345 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:15:44.345 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-22T13:15:44.372 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-22T13:15:44.439 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:15:44.439 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-22T13:15:44.468 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-22T13:15:44.536 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-22T13:15:44.537 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:15:44.537 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/stdin-killer 2023-09-22T13:15:44.564 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-22T13:15:44.631 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:15:44.631 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/stdin-killer 2023-09-22T13:15:44.658 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-22T13:15:44.726 INFO:teuthology.run_tasks:Running task ceph... 2023-09-22T13:15:44.786 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-22T13:15:44.787 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 777 /var/log/ceph 2023-09-22T13:15:44.789 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 777 /var/log/ceph 2023-09-22T13:15:44.815 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-22T13:15:44.816 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-22T13:15:44.856 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-22T13:15:44.882 INFO:tasks.ceph:Creating extra log directories... 2023-09-22T13:15:44.882 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-22T13:15:44.923 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-22T13:15:44.961 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-22T13:15:44.961 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10, 'rgw max chunk size': 4194304, 'rgw obj stripe size': 6291456, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000, 'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'lockdep': True}, 'osd': {'debug ms': 1, 'debug osd': 20, 'lockdep': True, 'osd objectstore': 'filestore', 'osd shutdown pgref assert': True, 'osd sloppy crc': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-22T13:15:44.961 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/cfsnyder-2023-09-22_11:28:34-rgw-wip-cfsnyder-bucket-check-pacific-distro-default-smithi/7400929', 'branch': 'wip-cfsnyder-bucket-check-pacific', 'description': 'rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7400929', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'cfsnyder-2023-09-22_11:28:34-rgw-wip-cfsnyder-bucket-check-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-cfsnyder-bucket-check-pacific'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10, 'rgw max chunk size': 4194304, 'rgw obj stripe size': 6291456, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000, 'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'lockdep': True}, 'osd': {'debug ms': 1, 'debug osd': 20, 'lockdep': True, 'osd objectstore': 'filestore', 'osd shutdown pgref assert': True, 'osd sloppy crc': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '31c126b7c1c0c06b66d9d19c5bfc51c46b503392'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '31c126b7c1c0c06b66d9d19c5bfc51c46b503392'}}, 'rgw': {'compression type': 'random', 'ec-data-pool': False, 'frontend': 'beast', 'storage classes': 'LUKEWARM, FROZEN'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-cfsnyder-bucket-check-pacific', 'sha1': '31c126b7c1c0c06b66d9d19c5bfc51c46b503392'}}, 'owner': 'scheduled_cfsnyder@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'sha1': '31c126b7c1c0c06b66d9d19c5bfc51c46b503392', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-cfsnyder-bucket-check-pacific', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_31c126b7c1c0c06b66d9d19c5bfc51c46b503392/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '31c126b7c1c0c06b66d9d19c5bfc51c46b503392', 'targets': {'smithi096.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPqB9cWhFW7jYHtdBsk4nvELU9+BPYn+lLgm5zcLstUd8oFjM39ksSWoLh+5MZWrJckhoXwLLGmdqHubeLuPskw=', 'smithi105.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEcE1zfd77YNZFbzP/oD1iV7nuNlVs4xW+tYZ1nrgTp5ZTad3T5piHy5PJibq2fKG/8lPvzIpja/Ox4ioWQ4Iq8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'openssl_keys': None}, {'rgw': {'client.0': None}}, {'workunit': {'clients': {'client.0': ['rgw/run-bucket-check.sh']}}}, {'workunit': {'clients': {'client.0': ['cls/test_cls_lock.sh', 'cls/test_cls_log.sh', 'cls/test_cls_refcount.sh', 'cls/test_cls_rgw.sh', 'cls/test_cls_cmpomap.sh', 'cls/test_cls_2pc_queue.sh', 'rgw/test_rgw_gc_log.sh', 'rgw/test_rgw_obj.sh', 'rgw/test_rgw_throttle.sh', 'rgw/test_librgw_file.sh']}}}, {'ragweed': {'client.0': {'default-branch': 'ceph-pacific', 'rgw_server': 'client.0', 'stages': 'prepare,check'}}}, {'workunit': {'clients': {'client.0': ['rgw/run-reshard.sh']}}}, {'s3tests-java': {'client.0': {'force-branch': 'ceph-pacific', 'force-repo': 'https://github.com/ceph/java_s3tests.git'}}}, {'s3tests': {'client.0': {'force-branch': 'ceph-pacific', 'rgw_server': 'client.0'}}}, {'workunit': {'clients': {'client.0': ['rgw/run-versioning.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-22_11:28:34', 'tube': 'smithi', 'user': 'cfsnyder', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-22T13:15:44.962 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-22T13:15:44.991 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-22T13:15:45.008 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/run/ceph 2023-09-22T13:15:45.048 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m0777 -- /var/run/ceph 2023-09-22T13:15:45.075 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:15:45.075 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/scratch_devs of=/dev/stdout 2023-09-22T13:15:45.131 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-22T13:15:45.131 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_1 2023-09-22T13:15:45.189 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-22T13:15:45.189 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-22T13:15:45.189 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 6h/6d Inode: 424965 Links: 1 2023-09-22T13:15:45.189 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-22T13:15:45.189 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2023-09-22T13:15:45.189 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2023-09-22 13:15:38.704871429 +0000 2023-09-22T13:15:45.190 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2023-09-22 12:20:34.381843080 +0000 2023-09-22T13:15:45.190 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2023-09-22 12:20:34.381843080 +0000 2023-09-22T13:15:45.190 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: - 2023-09-22T13:15:45.190 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-22T13:15:45.255 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2023-09-22T13:15:45.255 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2023-09-22T13:15:45.255 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.00020283 s, 2.5 MB/s 2023-09-22T13:15:45.256 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-22T13:15:45.315 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_2 2023-09-22T13:15:45.372 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-22T13:15:45.373 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-22T13:15:45.373 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 6h/6d Inode: 423170 Links: 1 2023-09-22T13:15:45.373 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-22T13:15:45.373 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2023-09-22T13:15:45.373 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2023-09-22 13:15:38.704871429 +0000 2023-09-22T13:15:45.373 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2023-09-22 12:20:34.941830439 +0000 2023-09-22T13:15:45.373 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2023-09-22 12:20:34.941830439 +0000 2023-09-22T13:15:45.373 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: - 2023-09-22T13:15:45.373 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-22T13:15:45.439 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2023-09-22T13:15:45.439 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2023-09-22T13:15:45.439 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000214247 s, 2.4 MB/s 2023-09-22T13:15:45.440 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-22T13:15:45.499 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_3 2023-09-22T13:15:45.556 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-22T13:15:45.556 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-22T13:15:45.557 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 6h/6d Inode: 422514 Links: 1 2023-09-22T13:15:45.557 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-22T13:15:45.557 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2023-09-22T13:15:45.557 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2023-09-22 13:15:38.704871429 +0000 2023-09-22T13:15:45.557 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2023-09-22 12:20:35.455818836 +0000 2023-09-22T13:15:45.557 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2023-09-22 12:20:35.455818836 +0000 2023-09-22T13:15:45.557 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: - 2023-09-22T13:15:45.557 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-22T13:15:45.622 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2023-09-22T13:15:45.622 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2023-09-22T13:15:45.623 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000214486 s, 2.4 MB/s 2023-09-22T13:15:45.623 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-22T13:15:45.683 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_4 2023-09-22T13:15:45.741 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-22T13:15:45.741 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-22T13:15:45.741 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 6h/6d Inode: 419332 Links: 1 2023-09-22T13:15:45.741 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-22T13:15:45.741 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2023-09-22T13:15:45.741 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2023-09-22 13:15:38.704871429 +0000 2023-09-22T13:15:45.741 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2023-09-22 12:20:35.959807459 +0000 2023-09-22T13:15:45.741 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2023-09-22 12:20:35.959807459 +0000 2023-09-22T13:15:45.741 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: - 2023-09-22T13:15:45.742 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-22T13:15:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2023-09-22T13:15:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2023-09-22T13:15:45.807 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000246951 s, 2.1 MB/s 2023-09-22T13:15:45.808 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-22T13:15:45.866 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-22T13:15:45.867 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:15:45.867 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2023-09-22T13:15:45.887 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-22T13:15:45.887 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 2023-09-22T13:15:45.947 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-22T13:15:45.947 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-22T13:15:45.947 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 6h/6d Inode: 417128 Links: 1 2023-09-22T13:15:45.947 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-22T13:15:45.947 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2023-09-22T13:15:45.947 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2023-09-22 13:14:28.174698372 +0000 2023-09-22T13:15:45.947 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2023-09-22 12:20:52.498538037 +0000 2023-09-22T13:15:45.948 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2023-09-22 12:20:52.498538037 +0000 2023-09-22T13:15:45.948 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2023-09-22T13:15:45.948 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-22T13:15:46.016 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2023-09-22T13:15:46.016 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2023-09-22T13:15:46.016 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000258406 s, 2.0 MB/s 2023-09-22T13:15:46.017 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-22T13:15:46.077 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 2023-09-22T13:15:46.136 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-22T13:15:46.136 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-22T13:15:46.136 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 6h/6d Inode: 417288 Links: 1 2023-09-22T13:15:46.136 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-22T13:15:46.136 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2023-09-22T13:15:46.136 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2023-09-22 13:14:28.174698372 +0000 2023-09-22T13:15:46.136 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2023-09-22 12:20:53.012528224 +0000 2023-09-22T13:15:46.137 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2023-09-22 12:20:53.012528224 +0000 2023-09-22T13:15:46.137 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2023-09-22T13:15:46.137 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-22T13:15:46.203 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2023-09-22T13:15:46.282 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2023-09-22T13:15:46.283 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000221273 s, 2.3 MB/s 2023-09-22T13:15:46.284 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-22T13:15:46.344 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 2023-09-22T13:15:46.402 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-22T13:15:46.403 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-22T13:15:46.403 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 6h/6d Inode: 420320 Links: 1 2023-09-22T13:15:46.403 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-22T13:15:46.403 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2023-09-22T13:15:46.403 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2023-09-22 13:14:28.174698372 +0000 2023-09-22T13:15:46.403 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2023-09-22 12:20:53.526518411 +0000 2023-09-22T13:15:46.403 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2023-09-22 12:20:53.526518411 +0000 2023-09-22T13:15:46.403 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2023-09-22T13:15:46.404 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-22T13:15:46.468 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2023-09-22T13:15:46.468 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2023-09-22T13:15:46.469 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000238004 s, 2.2 MB/s 2023-09-22T13:15:46.469 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-22T13:15:46.528 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 2023-09-22T13:15:46.586 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-22T13:15:46.586 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-22T13:15:46.586 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 6h/6d Inode: 424273 Links: 1 2023-09-22T13:15:46.586 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-22T13:15:46.587 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2023-09-22T13:15:46.587 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2023-09-22 13:14:28.174698372 +0000 2023-09-22T13:15:46.587 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2023-09-22 12:20:54.026508865 +0000 2023-09-22T13:15:46.587 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2023-09-22 12:20:54.026508865 +0000 2023-09-22T13:15:46.587 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2023-09-22T13:15:46.587 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-22T13:15:46.651 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2023-09-22T13:15:46.651 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2023-09-22T13:15:46.651 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000251121 s, 2.0 MB/s 2023-09-22T13:15:46.652 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-22T13:15:46.711 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-22T13:15:46.711 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi105.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-22T13:15:46.711 INFO:tasks.ceph:Generating config... 2023-09-22T13:15:46.712 INFO:tasks.ceph:[client] debug rgw = 20 2023-09-22T13:15:46.712 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-09-22T13:15:46.713 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-09-22T13:15:46.713 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2023-09-22T13:15:46.713 INFO:tasks.ceph:[client] rgw lc debug interval = 10 2023-09-22T13:15:46.713 INFO:tasks.ceph:[client] rgw max chunk size = 4194304 2023-09-22T13:15:46.713 INFO:tasks.ceph:[client] rgw obj stripe size = 6291456 2023-09-22T13:15:46.713 INFO:tasks.ceph:[client] setgroup = ceph 2023-09-22T13:15:46.713 INFO:tasks.ceph:[client] setuser = ceph 2023-09-22T13:15:46.713 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-09-22T13:15:46.713 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-09-22T13:15:46.713 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 10 2023-09-22T13:15:46.714 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 10 2023-09-22T13:15:46.714 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-22T13:15:46.714 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-22T13:15:46.714 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-22T13:15:46.714 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-22T13:15:46.714 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-22T13:15:46.714 INFO:tasks.ceph:[mon] lockdep = True 2023-09-22T13:15:46.714 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-22T13:15:46.714 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-22T13:15:46.714 INFO:tasks.ceph:[osd] lockdep = True 2023-09-22T13:15:46.714 INFO:tasks.ceph:[osd] osd objectstore = filestore 2023-09-22T13:15:46.715 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-09-22T13:15:46.715 INFO:tasks.ceph:[osd] osd sloppy crc = True 2023-09-22T13:15:46.715 INFO:tasks.ceph:Setting up mon.a... 2023-09-22T13:15:46.715 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-22T13:15:46.823 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/ceph.keyring 2023-09-22T13:15:46.864 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-22T13:15:46.909 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-22T13:15:46.936 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.96'), ('mon.c', '[v2:172.21.15.96:3301,v1:172.21.15.96:6790]'), ('mon.b', '172.21.15.105')] 2023-09-22T13:15:46.937 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.96,[v2:172.21.15.96:3301,v1:172.21.15.96:6790],172.21.15.105', 'mon client directed command retry': 5, 'ms inject socket failures': 5000, 'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'lockdep': True, 'osd objectstore': 'filestore', 'osd shutdown pgref assert': True, 'osd sloppy crc': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'lockdep': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10, 'rgw max chunk size': 4194304, 'rgw obj stripe size': 6291456, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-09-22T13:15:46.938 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:15:46.938 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-22T13:15:46.995 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.96 --addv c '[v2:172.21.15.96:3301,v1:172.21.15.96:6790]' --add b 172.21.15.105 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-22T13:15:47.082 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:15:47.082 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:15:47.082 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-22T13:15:47.083 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: generated fsid 37ee0465-b78b-4c40-811f-70b4ab3c2f62 2023-09-22T13:15:47.083 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 0 2023-09-22T13:15:47.083 INFO:teuthology.orchestra.run.smithi096.stdout:fsid 37ee0465-b78b-4c40-811f-70b4ab3c2f62 2023-09-22T13:15:47.083 INFO:teuthology.orchestra.run.smithi096.stdout:last_changed 2023-09-22T13:15:47.083360+0000 2023-09-22T13:15:47.083 INFO:teuthology.orchestra.run.smithi096.stdout:created 2023-09-22T13:15:47.083360+0000 2023-09-22T13:15:47.083 INFO:teuthology.orchestra.run.smithi096.stdout:min_mon_release 0 (unknown) 2023-09-22T13:15:47.083 INFO:teuthology.orchestra.run.smithi096.stdout:election_strategy: 1 2023-09-22T13:15:47.083 INFO:teuthology.orchestra.run.smithi096.stdout:0: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.a 2023-09-22T13:15:47.083 INFO:teuthology.orchestra.run.smithi096.stdout:1: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.b 2023-09-22T13:15:47.083 INFO:teuthology.orchestra.run.smithi096.stdout:2: [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] mon.c 2023-09-22T13:15:47.083 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-09-22T13:15:47.084 DEBUG:teuthology.orchestra.run.smithi096:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-22T13:15:47.102 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 37ee0465-b78b-4c40-811f-70b4ab3c2f62... 2023-09-22T13:15:47.103 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-22T13:15:47.144 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-22T13:15:47.187 INFO:teuthology.orchestra.run.smithi096.stdout:[global] 2023-09-22T13:15:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: chdir = "" 2023-09-22T13:15:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-22T13:15:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: auth supported = cephx 2023-09-22T13:15:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: filestore xattr use omap = true 2023-09-22T13:15:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: mon clock drift allowed = 1.000 2023-09-22T13:15:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: osd crush chooseleaf type = 0 2023-09-22T13:15:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: auth debug = true 2023-09-22T13:15:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on old message = true 2023-09-22T13:15:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on bug = true 2023-09-22T13:15:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-22T13:15:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: mon pg warn max object skew = 0 2023-09-22T13:15:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: # disable pg_autoscaler by default for new pools 2023-09-22T13:15:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-22T13:15:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default size = 2 2023-09-22T13:15:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow primary affinity = true 2023-09-22T13:15:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow pg remap = true 2023-09-22T13:15:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on legacy crush tunables = false 2023-09-22T13:15:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on crush straw calc version zero = false 2023-09-22T13:15:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on no sortbitwise = false 2023-09-22T13:15:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on osd down out interval zero = false 2023-09-22T13:15:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on too few osds = false 2023-09-22T13:15:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-22T13:15:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-22T13:15:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: mon_allow_pool_size_one = true 2023-09-22T13:15:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-09-22T13:15:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: osd default data pool replay window = 5 2023-09-22T13:15:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: mon allow pool delete = true 2023-09-22T13:15:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: mon cluster log file level = debug 2023-09-22T13:15:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: debug asserts on shutdown = true 2023-09-22T13:15:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: mon health detail to clog = false 2023-09-22T13:15:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: mon host = "172.21.15.96,[v2:172.21.15.96:3301,v1:172.21.15.96:6790],172.21.15.105" 2023-09-22T13:15:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: mon client directed command retry = 5 2023-09-22T13:15:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: ms inject socket failures = 5000 2023-09-22T13:15:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: osd_max_pg_log_entries = 10 2023-09-22T13:15:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: osd_min_pg_log_entries = 10 2023-09-22T13:15:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: fsid = 37ee0465-b78b-4c40-811f-70b4ab3c2f62 2023-09-22T13:15:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.191 INFO:teuthology.orchestra.run.smithi096.stdout:[osd] 2023-09-22T13:15:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: osd journal size = 100 2023-09-22T13:15:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub load threshold = 5.0 2023-09-22T13:15:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub max interval = 600 2023-09-22T13:15:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: osd recover clone overlap = true 2023-09-22T13:15:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: osd recovery max chunk = 1048576 2023-09-22T13:15:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug shutdown = true 2023-09-22T13:15:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug op order = true 2023-09-22T13:15:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug verify stray on activate = true 2023-09-22T13:15:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: osd open classes on start = true 2023-09-22T13:15:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug pg log writeout = true 2023-09-22T13:15:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: osd deep scrub update digest min age = 30 2023-09-22T13:15:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: osd map max advance = 10 2023-09-22T13:15:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: journal zero on create = true 2023-09-22T13:15:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: filestore ondisk finisher threads = 3 2023-09-22T13:15:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: filestore apply finisher threads = 3 2023-09-22T13:15:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: bdev debug aio = true 2023-09-22T13:15:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug misdirected ops = true 2023-09-22T13:15:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2023-09-22T13:15:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: debug osd = 20 2023-09-22T13:15:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: lockdep = True 2023-09-22T13:15:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: osd objectstore = filestore 2023-09-22T13:15:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: osd shutdown pgref assert = True 2023-09-22T13:15:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: osd sloppy crc = True 2023-09-22T13:15:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.194 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr] 2023-09-22T13:15:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2023-09-22T13:15:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: debug mgr = 20 2023-09-22T13:15:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2023-09-22T13:15:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2023-09-22T13:15:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2023-09-22T13:15:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2023-09-22T13:15:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: mgr/telemetry/nag = false 2023-09-22T13:15:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.195 INFO:teuthology.orchestra.run.smithi096.stdout:[mon] 2023-09-22T13:15:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2023-09-22T13:15:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2023-09-22T13:15:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: debug paxos = 20 2023-09-22T13:15:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2023-09-22T13:15:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: mon data avail warn = 5 2023-09-22T13:15:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: mon mgr mkfs grace = 240 2023-09-22T13:15:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2023-09-22T13:15:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd reporter subtree level = osd 2023-09-22T13:15:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd prime pg temp = true 2023-09-22T13:15:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2023-09-22T13:15:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-22T13:15:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: auth mon ticket ttl = 660 # 11m 2023-09-22T13:15:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: auth service ticket ttl = 240 # 4m 2023-09-22T13:15:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: # don't complain about insecure global_id in the test suite 2023-09-22T13:15:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-22T13:15:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-22T13:15:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: lockdep = True 2023-09-22T13:15:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:15:47.196 INFO:teuthology.orchestra.run.smithi096.stdout:[client] 2023-09-22T13:15:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: rgw cache enabled = true 2023-09-22T13:15:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable ops log = true 2023-09-22T13:15:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable usage log = true 2023-09-22T13:15:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-22T13:15:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-22T13:15:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: debug rgw = 20 2023-09-22T13:15:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: rgw crypt require ssl = False 2023-09-22T13:15:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: rgw crypt s3 kms backend = testing 2023-09-22T13:15:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2023-09-22T13:15:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: rgw lc debug interval = 10 2023-09-22T13:15:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: rgw max chunk size = 4194304 2023-09-22T13:15:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: rgw obj stripe size = 6291456 2023-09-22T13:15:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: setgroup = ceph 2023-09-22T13:15:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: setuser = ceph 2023-09-22T13:15:47.197 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.a] 2023-09-22T13:15:47.198 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.c] 2023-09-22T13:15:47.198 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.b] 2023-09-22T13:15:47.198 INFO:teuthology.orchestra.run.smithi105.stdout:[global] 2023-09-22T13:15:47.198 INFO:teuthology.orchestra.run.smithi105.stdout: chdir = "" 2023-09-22T13:15:47.198 INFO:teuthology.orchestra.run.smithi105.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-22T13:15:47.198 INFO:teuthology.orchestra.run.smithi105.stdout: auth supported = cephx 2023-09-22T13:15:47.199 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.199 INFO:teuthology.orchestra.run.smithi105.stdout: filestore xattr use omap = true 2023-09-22T13:15:47.199 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.199 INFO:teuthology.orchestra.run.smithi105.stdout: mon clock drift allowed = 1.000 2023-09-22T13:15:47.199 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.199 INFO:teuthology.orchestra.run.smithi105.stdout: osd crush chooseleaf type = 0 2023-09-22T13:15:47.199 INFO:teuthology.orchestra.run.smithi105.stdout: auth debug = true 2023-09-22T13:15:47.199 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.199 INFO:teuthology.orchestra.run.smithi105.stdout: ms die on old message = true 2023-09-22T13:15:47.199 INFO:teuthology.orchestra.run.smithi105.stdout: ms die on bug = true 2023-09-22T13:15:47.199 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.199 INFO:teuthology.orchestra.run.smithi105.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-22T13:15:47.200 INFO:teuthology.orchestra.run.smithi105.stdout: mon pg warn max object skew = 0 2023-09-22T13:15:47.200 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.200 INFO:teuthology.orchestra.run.smithi105.stdout: # disable pg_autoscaler by default for new pools 2023-09-22T13:15:47.200 INFO:teuthology.orchestra.run.smithi105.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-22T13:15:47.200 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.200 INFO:teuthology.orchestra.run.smithi105.stdout: osd pool default size = 2 2023-09-22T13:15:47.200 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.200 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd allow primary affinity = true 2023-09-22T13:15:47.200 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd allow pg remap = true 2023-09-22T13:15:47.200 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on legacy crush tunables = false 2023-09-22T13:15:47.200 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on crush straw calc version zero = false 2023-09-22T13:15:47.200 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on no sortbitwise = false 2023-09-22T13:15:47.200 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on osd down out interval zero = false 2023-09-22T13:15:47.201 INFO:teuthology.orchestra.run.smithi105.stdout: mon warn on too few osds = false 2023-09-22T13:15:47.201 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-22T13:15:47.201 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-22T13:15:47.201 INFO:teuthology.orchestra.run.smithi105.stdout: mon_allow_pool_size_one = true 2023-09-22T13:15:47.201 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.201 INFO:teuthology.orchestra.run.smithi105.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-09-22T13:15:47.201 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.201 INFO:teuthology.orchestra.run.smithi105.stdout: osd default data pool replay window = 5 2023-09-22T13:15:47.201 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.201 INFO:teuthology.orchestra.run.smithi105.stdout: mon allow pool delete = true 2023-09-22T13:15:47.201 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.201 INFO:teuthology.orchestra.run.smithi105.stdout: mon cluster log file level = debug 2023-09-22T13:15:47.202 INFO:teuthology.orchestra.run.smithi105.stdout: debug asserts on shutdown = true 2023-09-22T13:15:47.202 INFO:teuthology.orchestra.run.smithi105.stdout: mon health detail to clog = false 2023-09-22T13:15:47.202 INFO:teuthology.orchestra.run.smithi105.stdout: mon host = "172.21.15.96,[v2:172.21.15.96:3301,v1:172.21.15.96:6790],172.21.15.105" 2023-09-22T13:15:47.202 INFO:teuthology.orchestra.run.smithi105.stdout: mon client directed command retry = 5 2023-09-22T13:15:47.202 INFO:teuthology.orchestra.run.smithi105.stdout: ms inject socket failures = 5000 2023-09-22T13:15:47.202 INFO:teuthology.orchestra.run.smithi105.stdout: osd_max_pg_log_entries = 10 2023-09-22T13:15:47.202 INFO:teuthology.orchestra.run.smithi105.stdout: osd_min_pg_log_entries = 10 2023-09-22T13:15:47.202 INFO:teuthology.orchestra.run.smithi105.stdout: fsid = 37ee0465-b78b-4c40-811f-70b4ab3c2f62 2023-09-22T13:15:47.202 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.202 INFO:teuthology.orchestra.run.smithi105.stdout:[osd] 2023-09-22T13:15:47.202 INFO:teuthology.orchestra.run.smithi105.stdout: osd journal size = 100 2023-09-22T13:15:47.203 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.203 INFO:teuthology.orchestra.run.smithi105.stdout: osd scrub load threshold = 5.0 2023-09-22T13:15:47.203 INFO:teuthology.orchestra.run.smithi105.stdout: osd scrub max interval = 600 2023-09-22T13:15:47.203 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.203 INFO:teuthology.orchestra.run.smithi105.stdout: osd recover clone overlap = true 2023-09-22T13:15:47.203 INFO:teuthology.orchestra.run.smithi105.stdout: osd recovery max chunk = 1048576 2023-09-22T13:15:47.203 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.203 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug shutdown = true 2023-09-22T13:15:47.203 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug op order = true 2023-09-22T13:15:47.203 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug verify stray on activate = true 2023-09-22T13:15:47.203 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.203 INFO:teuthology.orchestra.run.smithi105.stdout: osd open classes on start = true 2023-09-22T13:15:47.203 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug pg log writeout = true 2023-09-22T13:15:47.204 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.204 INFO:teuthology.orchestra.run.smithi105.stdout: osd deep scrub update digest min age = 30 2023-09-22T13:15:47.204 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.204 INFO:teuthology.orchestra.run.smithi105.stdout: osd map max advance = 10 2023-09-22T13:15:47.204 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.204 INFO:teuthology.orchestra.run.smithi105.stdout: journal zero on create = true 2023-09-22T13:15:47.204 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.204 INFO:teuthology.orchestra.run.smithi105.stdout: filestore ondisk finisher threads = 3 2023-09-22T13:15:47.204 INFO:teuthology.orchestra.run.smithi105.stdout: filestore apply finisher threads = 3 2023-09-22T13:15:47.204 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.204 INFO:teuthology.orchestra.run.smithi105.stdout: bdev debug aio = true 2023-09-22T13:15:47.204 INFO:teuthology.orchestra.run.smithi105.stdout: osd debug misdirected ops = true 2023-09-22T13:15:47.205 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2023-09-22T13:15:47.205 INFO:teuthology.orchestra.run.smithi105.stdout: debug osd = 20 2023-09-22T13:15:47.205 INFO:teuthology.orchestra.run.smithi105.stdout: lockdep = True 2023-09-22T13:15:47.205 INFO:teuthology.orchestra.run.smithi105.stdout: osd objectstore = filestore 2023-09-22T13:15:47.205 INFO:teuthology.orchestra.run.smithi105.stdout: osd shutdown pgref assert = True 2023-09-22T13:15:47.205 INFO:teuthology.orchestra.run.smithi105.stdout: osd sloppy crc = True 2023-09-22T13:15:47.205 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.205 INFO:teuthology.orchestra.run.smithi105.stdout:[mgr] 2023-09-22T13:15:47.205 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2023-09-22T13:15:47.205 INFO:teuthology.orchestra.run.smithi105.stdout: debug mgr = 20 2023-09-22T13:15:47.205 INFO:teuthology.orchestra.run.smithi105.stdout: debug mon = 20 2023-09-22T13:15:47.205 INFO:teuthology.orchestra.run.smithi105.stdout: debug auth = 20 2023-09-22T13:15:47.206 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min pgs per osd = 4 2023-09-22T13:15:47.206 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min bytes per osd = 10 2023-09-22T13:15:47.206 INFO:teuthology.orchestra.run.smithi105.stdout: mgr/telemetry/nag = false 2023-09-22T13:15:47.206 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.206 INFO:teuthology.orchestra.run.smithi105.stdout:[mon] 2023-09-22T13:15:47.206 INFO:teuthology.orchestra.run.smithi105.stdout: debug ms = 1 2023-09-22T13:15:47.206 INFO:teuthology.orchestra.run.smithi105.stdout: debug mon = 20 2023-09-22T13:15:47.206 INFO:teuthology.orchestra.run.smithi105.stdout: debug paxos = 20 2023-09-22T13:15:47.206 INFO:teuthology.orchestra.run.smithi105.stdout: debug auth = 20 2023-09-22T13:15:47.206 INFO:teuthology.orchestra.run.smithi105.stdout: mon data avail warn = 5 2023-09-22T13:15:47.206 INFO:teuthology.orchestra.run.smithi105.stdout: mon mgr mkfs grace = 240 2023-09-22T13:15:47.206 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min pgs per osd = 4 2023-09-22T13:15:47.207 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd reporter subtree level = osd 2023-09-22T13:15:47.207 INFO:teuthology.orchestra.run.smithi105.stdout: mon osd prime pg temp = true 2023-09-22T13:15:47.207 INFO:teuthology.orchestra.run.smithi105.stdout: mon reweight min bytes per osd = 10 2023-09-22T13:15:47.207 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.207 INFO:teuthology.orchestra.run.smithi105.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-22T13:15:47.207 INFO:teuthology.orchestra.run.smithi105.stdout: auth mon ticket ttl = 660 # 11m 2023-09-22T13:15:47.207 INFO:teuthology.orchestra.run.smithi105.stdout: auth service ticket ttl = 240 # 4m 2023-09-22T13:15:47.207 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.207 INFO:teuthology.orchestra.run.smithi105.stdout: # don't complain about insecure global_id in the test suite 2023-09-22T13:15:47.207 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-22T13:15:47.207 INFO:teuthology.orchestra.run.smithi105.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-22T13:15:47.207 INFO:teuthology.orchestra.run.smithi105.stdout: lockdep = True 2023-09-22T13:15:47.208 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-09-22T13:15:47.208 INFO:teuthology.orchestra.run.smithi105.stdout:[client] 2023-09-22T13:15:47.208 INFO:teuthology.orchestra.run.smithi105.stdout: rgw cache enabled = true 2023-09-22T13:15:47.208 INFO:teuthology.orchestra.run.smithi105.stdout: rgw enable ops log = true 2023-09-22T13:15:47.208 INFO:teuthology.orchestra.run.smithi105.stdout: rgw enable usage log = true 2023-09-22T13:15:47.208 INFO:teuthology.orchestra.run.smithi105.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-22T13:15:47.208 INFO:teuthology.orchestra.run.smithi105.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-22T13:15:47.208 INFO:teuthology.orchestra.run.smithi105.stdout: debug rgw = 20 2023-09-22T13:15:47.208 INFO:teuthology.orchestra.run.smithi105.stdout: rgw crypt require ssl = False 2023-09-22T13:15:47.208 INFO:teuthology.orchestra.run.smithi105.stdout: rgw crypt s3 kms backend = testing 2023-09-22T13:15:47.208 INFO:teuthology.orchestra.run.smithi105.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2023-09-22T13:15:47.208 INFO:teuthology.orchestra.run.smithi105.stdout: rgw lc debug interval = 10 2023-09-22T13:15:47.208 INFO:teuthology.orchestra.run.smithi105.stdout: rgw max chunk size = 4194304 2023-09-22T13:15:47.209 INFO:teuthology.orchestra.run.smithi105.stdout: rgw obj stripe size = 6291456 2023-09-22T13:15:47.209 INFO:teuthology.orchestra.run.smithi105.stdout: setgroup = ceph 2023-09-22T13:15:47.209 INFO:teuthology.orchestra.run.smithi105.stdout: setuser = ceph 2023-09-22T13:15:47.209 INFO:teuthology.orchestra.run.smithi105.stdout:[mon.a] 2023-09-22T13:15:47.209 INFO:teuthology.orchestra.run.smithi105.stdout:[mon.c] 2023-09-22T13:15:47.209 INFO:teuthology.orchestra.run.smithi105.stdout:[mon.b] 2023-09-22T13:15:47.210 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-22T13:15:47.210 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-22T13:15:47.302 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-22T13:15:47.302 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:15:47.303 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-22T13:15:47.320 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:15:47.320 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-22T13:15:47.379 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi096.front.sepia.ceph.com 2023-09-22T13:15:47.379 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:15:47.379 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-22T13:15:47.379 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-22T13:15:47.456 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:15:47.456 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-22T13:15:47.514 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi105.front.sepia.ceph.com 2023-09-22T13:15:47.514 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:15:47.514 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-22T13:15:47.514 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-22T13:15:47.551 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:15:47.551 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-22T13:15:47.609 INFO:tasks.ceph:Setting up mon nodes... 2023-09-22T13:15:47.609 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-22T13:15:47.609 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-09-22T13:15:47.658 INFO:teuthology.orchestra.run.smithi096.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-09-22T13:15:47.660 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-22T13:15:47.775 INFO:teuthology.orchestra.run.smithi105.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-22T13:15:47.777 INFO:tasks.ceph:Setting up mds nodes... 2023-09-22T13:15:47.777 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-22T13:15:47.778 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-22T13:15:47.817 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-22T13:15:47.829 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-09-22T13:15:47.869 INFO:teuthology.orchestra.run.smithi105.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-09-22T13:15:47.882 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-22T13:15:47.882 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi105.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-22T13:15:47.882 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-22T13:15:47.909 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-22T13:15:47.910 INFO:tasks.ceph:role: osd.0 2023-09-22T13:15:47.910 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com 2023-09-22T13:15:47.910 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-22T13:15:47.985 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-22T13:15:47.989 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-22T13:15:47.989 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-22T13:15:47.990 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-22T13:15:47.990 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-22T13:15:47.990 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2023-09-22T13:15:47.990 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-22T13:15:47.990 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-22T13:15:47.990 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-22T13:15:47.990 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-22T13:15:48.420 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2023-09-22T13:15:48.422 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2023-09-22T13:15:48.422 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-22T13:15:48.583 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-22T13:15:48.617 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-22T13:15:48.685 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-22T13:15:48.685 INFO:tasks.ceph:role: osd.1 2023-09-22T13:15:48.685 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com 2023-09-22T13:15:48.685 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-22T13:15:48.759 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-22T13:15:48.759 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-22T13:15:48.759 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-22T13:15:48.759 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-22T13:15:48.759 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-22T13:15:48.759 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2023-09-22T13:15:48.759 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-22T13:15:48.759 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-22T13:15:48.760 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-22T13:15:48.760 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-22T13:15:49.188 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2023-09-22T13:15:49.189 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2023-09-22T13:15:49.190 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-22T13:15:49.296 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-22T13:15:49.327 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-22T13:15:49.395 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-22T13:15:49.395 INFO:tasks.ceph:role: osd.2 2023-09-22T13:15:49.395 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com 2023-09-22T13:15:49.395 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-22T13:15:49.470 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-22T13:15:49.470 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-22T13:15:49.470 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-22T13:15:49.470 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-22T13:15:49.470 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-22T13:15:49.470 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2023-09-22T13:15:49.470 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-22T13:15:49.470 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-22T13:15:49.471 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-22T13:15:49.471 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-22T13:15:49.927 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2023-09-22T13:15:49.929 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2023-09-22T13:15:49.929 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-22T13:15:50.096 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-22T13:15:50.127 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-09-22T13:15:50.194 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-22T13:15:50.194 INFO:tasks.ceph:role: osd.3 2023-09-22T13:15:50.194 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi096.front.sepia.ceph.com 2023-09-22T13:15:50.195 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-22T13:15:50.269 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-22T13:15:50.269 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-22T13:15:50.269 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-22T13:15:50.269 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-22T13:15:50.270 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-22T13:15:50.270 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2023-09-22T13:15:50.270 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-22T13:15:50.270 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-22T13:15:50.270 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-22T13:15:50.270 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-22T13:15:50.698 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2023-09-22T13:15:50.700 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2023-09-22T13:15:50.700 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-09-22T13:15:50.870 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-09-22T13:15:50.904 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-22T13:15:50.997 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:50.997+0000 7ff0fc834200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-22T13:15:50.998 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:50.997+0000 7ff0fc834200 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-22T13:15:51.007 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.007+0000 7ff0fc834200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:15:51.007 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.007+0000 7ff0fc834200 -1 journal FileJournal::_open_file : zeroing journal 2023-09-22T13:15:51.198 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.198+0000 7ff0fc834200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:15:51.198 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.198+0000 7ff0fc834200 -1 journal do_read_entry(4096): bad header magic 2023-09-22T13:15:51.199 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.198+0000 7ff0fc834200 -1 journal do_read_entry(4096): bad header magic 2023-09-22T13:15:51.223 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-22T13:15:51.250 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-22T13:15:51.334 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.333+0000 7fcf2ea07200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-22T13:15:51.334 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.333+0000 7fcf2ea07200 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-22T13:15:51.338 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.338+0000 7fcf2ea07200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:15:51.338 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.338+0000 7fcf2ea07200 -1 journal FileJournal::_open_file : zeroing journal 2023-09-22T13:15:51.502 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.502+0000 7fcf2ea07200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:15:51.502 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.502+0000 7fcf2ea07200 -1 journal do_read_entry(4096): bad header magic 2023-09-22T13:15:51.503 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.502+0000 7fcf2ea07200 -1 journal do_read_entry(4096): bad header magic 2023-09-22T13:15:51.510 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-22T13:15:51.579 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-22T13:15:51.661 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.661+0000 7f0065981200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-22T13:15:51.661 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.661+0000 7f0065981200 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-22T13:15:51.666 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.666+0000 7f0065981200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:15:51.666 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.666+0000 7f0065981200 -1 journal FileJournal::_open_file : zeroing journal 2023-09-22T13:15:51.802 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.802+0000 7f0065981200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:15:51.802 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.802+0000 7f0065981200 -1 journal do_read_entry(4096): bad header magic 2023-09-22T13:15:51.802 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.802+0000 7f0065981200 -1 journal do_read_entry(4096): bad header magic 2023-09-22T13:15:51.812 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-22T13:15:51.879 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-22T13:15:51.961 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.961+0000 7f112e5b9200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-09-22T13:15:51.962 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.961+0000 7f112e5b9200 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-09-22T13:15:51.968 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.968+0000 7f112e5b9200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:15:51.968 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:51.968+0000 7f112e5b9200 -1 journal FileJournal::_open_file : zeroing journal 2023-09-22T13:15:52.093 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:52.093+0000 7f112e5b9200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:15:52.093 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:52.093+0000 7f112e5b9200 -1 journal do_read_entry(4096): bad header magic 2023-09-22T13:15:52.093 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:15:52.093+0000 7f112e5b9200 -1 journal do_read_entry(4096): bad header magic 2023-09-22T13:15:52.103 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-09-22T13:15:52.170 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-09-22T13:15:52.204 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-22T13:15:52.204 INFO:tasks.ceph:role: osd.4 2023-09-22T13:15:52.204 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi105.front.sepia.ceph.com 2023-09-22T13:15:52.204 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-22T13:15:52.282 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-22T13:15:52.283 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-22T13:15:52.283 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-22T13:15:52.283 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-22T13:15:52.283 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-22T13:15:52.283 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2023-09-22T13:15:52.283 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-22T13:15:52.283 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-22T13:15:52.283 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-22T13:15:52.283 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-22T13:15:52.720 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2023-09-22T13:15:52.722 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2023-09-22T13:15:52.722 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-09-22T13:15:52.898 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-09-22T13:15:52.937 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-09-22T13:15:53.007 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-22T13:15:53.008 INFO:tasks.ceph:role: osd.5 2023-09-22T13:15:53.008 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi105.front.sepia.ceph.com 2023-09-22T13:15:53.008 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-22T13:15:53.084 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-22T13:15:53.084 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-22T13:15:53.084 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-22T13:15:53.084 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-22T13:15:53.084 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-22T13:15:53.084 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2023-09-22T13:15:53.084 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-22T13:15:53.084 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-22T13:15:53.084 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-22T13:15:53.085 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-22T13:15:53.506 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2023-09-22T13:15:53.507 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2023-09-22T13:15:53.508 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-09-22T13:15:53.616 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-09-22T13:15:53.649 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-09-22T13:15:53.717 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-22T13:15:53.717 INFO:tasks.ceph:role: osd.6 2023-09-22T13:15:53.718 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi105.front.sepia.ceph.com 2023-09-22T13:15:53.718 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-22T13:15:53.792 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-22T13:15:53.792 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-22T13:15:53.792 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-22T13:15:53.792 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-22T13:15:53.792 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-22T13:15:53.792 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2023-09-22T13:15:53.792 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-22T13:15:53.792 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-22T13:15:53.792 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-22T13:15:53.793 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-22T13:15:54.238 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2023-09-22T13:15:54.239 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2023-09-22T13:15:54.239 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-09-22T13:15:54.403 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-09-22T13:15:54.435 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-09-22T13:15:54.503 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-22T13:15:54.503 INFO:tasks.ceph:role: osd.7 2023-09-22T13:15:54.503 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi105.front.sepia.ceph.com 2023-09-22T13:15:54.503 DEBUG:teuthology.orchestra.run.smithi105:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-22T13:15:54.578 INFO:teuthology.orchestra.run.smithi105.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-22T13:15:54.578 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-22T13:15:54.578 INFO:teuthology.orchestra.run.smithi105.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-22T13:15:54.579 INFO:teuthology.orchestra.run.smithi105.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-22T13:15:54.579 INFO:teuthology.orchestra.run.smithi105.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-22T13:15:54.579 INFO:teuthology.orchestra.run.smithi105.stdout: = sunit=0 swidth=0 blks 2023-09-22T13:15:54.579 INFO:teuthology.orchestra.run.smithi105.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-22T13:15:54.579 INFO:teuthology.orchestra.run.smithi105.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-22T13:15:54.579 INFO:teuthology.orchestra.run.smithi105.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-22T13:15:54.579 INFO:teuthology.orchestra.run.smithi105.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-22T13:15:55.016 INFO:teuthology.orchestra.run.smithi105.stdout:Discarding blocks...Done. 2023-09-22T13:15:55.017 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi105.front.sepia.ceph.com -o noatime 2023-09-22T13:15:55.018 DEBUG:teuthology.orchestra.run.smithi105:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-09-22T13:15:55.182 DEBUG:teuthology.orchestra.run.smithi105:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-09-22T13:15:55.213 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-22T13:15:55.314 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:55.313+0000 7fdd6c198200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-09-22T13:15:55.314 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:55.314+0000 7fdd6c198200 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-09-22T13:15:55.322 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:55.321+0000 7fdd6c198200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:15:55.322 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:55.321+0000 7fdd6c198200 -1 journal FileJournal::_open_file : zeroing journal 2023-09-22T13:15:55.530 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:55.529+0000 7fdd6c198200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:15:55.530 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:55.529+0000 7fdd6c198200 -1 journal do_read_entry(4096): bad header magic 2023-09-22T13:15:55.530 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:55.529+0000 7fdd6c198200 -1 journal do_read_entry(4096): bad header magic 2023-09-22T13:15:55.555 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-09-22T13:15:55.583 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-22T13:15:55.666 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:55.665+0000 7fee39a99200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-09-22T13:15:55.666 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:55.665+0000 7fee39a99200 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-09-22T13:15:55.670 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:55.670+0000 7fee39a99200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:15:55.671 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:55.670+0000 7fee39a99200 -1 journal FileJournal::_open_file : zeroing journal 2023-09-22T13:15:55.844 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:55.844+0000 7fee39a99200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:15:55.844 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:55.844+0000 7fee39a99200 -1 journal do_read_entry(4096): bad header magic 2023-09-22T13:15:55.844 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:55.844+0000 7fee39a99200 -1 journal do_read_entry(4096): bad header magic 2023-09-22T13:15:55.852 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-09-22T13:15:55.919 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-22T13:15:56.002 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:56.002+0000 7f31bbc5d200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-09-22T13:15:56.002 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:56.002+0000 7f31bbc5d200 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-09-22T13:15:56.007 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:56.007+0000 7f31bbc5d200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:15:56.007 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:56.007+0000 7f31bbc5d200 -1 journal FileJournal::_open_file : zeroing journal 2023-09-22T13:15:56.142 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:56.142+0000 7f31bbc5d200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:15:56.143 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:56.142+0000 7f31bbc5d200 -1 journal do_read_entry(4096): bad header magic 2023-09-22T13:15:56.143 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:56.142+0000 7f31bbc5d200 -1 journal do_read_entry(4096): bad header magic 2023-09-22T13:15:56.152 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-09-22T13:15:56.220 DEBUG:teuthology.orchestra.run.smithi105:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-22T13:15:56.303 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:56.302+0000 7fe9b76ae200 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-09-22T13:15:56.303 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:56.303+0000 7fe9b76ae200 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-09-22T13:15:56.310 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:56.309+0000 7fe9b76ae200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:15:56.310 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:56.309+0000 7fe9b76ae200 -1 journal FileJournal::_open_file : zeroing journal 2023-09-22T13:15:56.456 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:56.456+0000 7fe9b76ae200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:15:56.517 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:56.456+0000 7fe9b76ae200 -1 journal do_read_entry(4096): bad header magic 2023-09-22T13:15:56.517 INFO:teuthology.orchestra.run.smithi105.stderr:2023-09-22T13:15:56.456+0000 7fe9b76ae200 -1 journal do_read_entry(4096): bad header magic 2023-09-22T13:15:56.518 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-09-22T13:15:56.586 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-22T13:15:56.586 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:15:56.586 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-09-22T13:15:56.619 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:15:56.620 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-22T13:15:56.691 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:15:56.691 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-22T13:15:56.761 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:15:56.762 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-22T13:15:56.831 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:15:56.832 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-09-22T13:15:56.901 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:15:56.901 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-22T13:15:56.929 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:15:56.929 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-09-22T13:15:56.996 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:15:56.997 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-09-22T13:15:57.063 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:15:57.063 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-09-22T13:15:57.130 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:15:57.130 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-09-22T13:15:57.197 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:15:57.197 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-22T13:15:57.216 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:15:57.217 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-09-22T13:15:57.255 INFO:tasks.ceph:Adding keys to all mons... 2023-09-22T13:15:57.255 DEBUG:teuthology.orchestra.run.smithi096:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-22T13:15:57.259 DEBUG:teuthology.orchestra.run.smithi105:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-22T13:15:57.298 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr.y] 2023-09-22T13:15:57.298 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCDkw1ldlpOJxAAGnifK6RQqu05G1nYlWfS1w== 2023-09-22T13:15:57.298 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.0] 2023-09-22T13:15:57.299 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCGkw1lGoaGOxAAOArjnu5WysM8bMtDrwGDfg== 2023-09-22T13:15:57.299 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.1] 2023-09-22T13:15:57.299 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCHkw1luubvExAAq3A2YLhWpk+oNG9wTuxk+g== 2023-09-22T13:15:57.299 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.2] 2023-09-22T13:15:57.299 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCHkw1lmh95JxAA/tQmujdINKecz74neUBROw== 2023-09-22T13:15:57.299 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.3] 2023-09-22T13:15:57.299 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCHkw1lG3hgORAAt/EVzS/fwCwxLg4I21OAFw== 2023-09-22T13:15:57.299 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr.x] 2023-09-22T13:15:57.299 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCDkw1l9C9FLhAAfKK5Gq72BU8sg3ePPfncZw== 2023-09-22T13:15:57.299 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.4] 2023-09-22T13:15:57.299 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCLkw1lRObKEhAACrQBBq0twuUl/VjkGJ1RQA== 2023-09-22T13:15:57.300 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.5] 2023-09-22T13:15:57.300 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCLkw1lkq7AJxAAwtayJ+7fMiMZ10e/gpjqhg== 2023-09-22T13:15:57.300 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.6] 2023-09-22T13:15:57.300 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCMkw1l0Wk1ABAAiLxqCO8sDGKxjSg4s77V6A== 2023-09-22T13:15:57.300 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.7] 2023-09-22T13:15:57.300 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCMkw1lgEUgEhAAmc4WGpfWhwPVjxxkFLJ6sA== 2023-09-22T13:15:57.300 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2023-09-22T13:15:57.300 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCDkw1ljgzOMBAA5Vzp5EjlOxbrfoKVXN9g3w== 2023-09-22T13:15:57.300 INFO:teuthology.orchestra.run.smithi096.stdout:[client.1] 2023-09-22T13:15:57.300 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCDkw1l2eDjMxAAlx3sKoAYjshDjJ34W8McMw== 2023-09-22T13:15:57.321 INFO:teuthology.orchestra.run.smithi105.stdout:[mgr.y] 2023-09-22T13:15:57.321 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQCDkw1ldlpOJxAAGnifK6RQqu05G1nYlWfS1w== 2023-09-22T13:15:57.321 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.0] 2023-09-22T13:15:57.321 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQCGkw1lGoaGOxAAOArjnu5WysM8bMtDrwGDfg== 2023-09-22T13:15:57.321 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.1] 2023-09-22T13:15:57.321 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQCHkw1luubvExAAq3A2YLhWpk+oNG9wTuxk+g== 2023-09-22T13:15:57.321 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.2] 2023-09-22T13:15:57.322 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQCHkw1lmh95JxAA/tQmujdINKecz74neUBROw== 2023-09-22T13:15:57.322 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.3] 2023-09-22T13:15:57.322 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQCHkw1lG3hgORAAt/EVzS/fwCwxLg4I21OAFw== 2023-09-22T13:15:57.322 INFO:teuthology.orchestra.run.smithi105.stdout:[mgr.x] 2023-09-22T13:15:57.322 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQCDkw1l9C9FLhAAfKK5Gq72BU8sg3ePPfncZw== 2023-09-22T13:15:57.322 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.4] 2023-09-22T13:15:57.322 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQCLkw1lRObKEhAACrQBBq0twuUl/VjkGJ1RQA== 2023-09-22T13:15:57.322 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.5] 2023-09-22T13:15:57.322 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQCLkw1lkq7AJxAAwtayJ+7fMiMZ10e/gpjqhg== 2023-09-22T13:15:57.322 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.6] 2023-09-22T13:15:57.322 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQCMkw1l0Wk1ABAAiLxqCO8sDGKxjSg4s77V6A== 2023-09-22T13:15:57.322 INFO:teuthology.orchestra.run.smithi105.stdout:[osd.7] 2023-09-22T13:15:57.323 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQCMkw1lgEUgEhAAmc4WGpfWhwPVjxxkFLJ6sA== 2023-09-22T13:15:57.323 INFO:teuthology.orchestra.run.smithi105.stdout:[client.0] 2023-09-22T13:15:57.323 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQCDkw1ljgzOMBAA5Vzp5EjlOxbrfoKVXN9g3w== 2023-09-22T13:15:57.323 INFO:teuthology.orchestra.run.smithi105.stdout:[client.1] 2023-09-22T13:15:57.323 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQCDkw1l2eDjMxAAlx3sKoAYjshDjJ34W8McMw== 2023-09-22T13:15:57.324 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-22T13:15:57.343 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-22T13:15:57.413 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T13:15:57.415 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T13:15:57.463 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T13:15:57.465 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T13:15:57.514 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T13:15:57.516 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T13:15:57.564 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T13:15:57.566 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T13:15:57.615 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-22T13:15:57.617 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-22T13:15:57.666 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T13:15:57.668 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T13:15:57.716 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T13:15:57.718 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T13:15:57.767 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T13:15:57.769 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T13:15:57.817 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T13:15:57.819 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-22T13:15:57.867 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-22T13:15:57.869 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-22T13:15:57.921 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-22T13:15:57.923 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-22T13:15:57.974 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-22T13:15:57.974 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-22T13:15:58.001 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-22T13:15:58.095 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-22T13:15:58.122 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-22T13:15:58.189 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-22T13:15:58.282 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-22T13:15:58.309 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-22T13:15:58.338 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-22T13:15:58.434 DEBUG:teuthology.orchestra.run.smithi105:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-22T13:15:58.461 DEBUG:teuthology.orchestra.run.smithi096:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-22T13:15:58.463 DEBUG:teuthology.orchestra.run.smithi105:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-22T13:15:58.519 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-22T13:15:58.519 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-22T13:15:58.519 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-22T13:15:58.521 INFO:tasks.ceph.mon.a:Started 2023-09-22T13:15:58.521 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-22T13:15:58.522 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-09-22T13:15:58.524 INFO:tasks.ceph.mon.c:Started 2023-09-22T13:15:58.524 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-22T13:15:58.524 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-09-22T13:15:58.561 INFO:tasks.ceph.mon.b:Started 2023-09-22T13:15:58.562 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-22T13:15:58.562 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-09-22T13:15:58.562 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-09-22T13:15:58.564 INFO:tasks.ceph.mgr.y:Started 2023-09-22T13:15:58.565 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-22T13:15:58.565 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-22T13:15:58.566 INFO:tasks.ceph.mgr.x:Started 2023-09-22T13:15:58.567 INFO:tasks.ceph:Setting crush tunables to default 2023-09-22T13:15:58.567 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd crush tunables default 2023-09-22T13:16:03.954 INFO:teuthology.orchestra.run.smithi096.stderr:adjusted tunables profile to default 2023-09-22T13:16:03.965 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-22T13:16:03.965 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:16:03.965 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-22T13:16:03.994 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:16:03.994 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-22T13:16:04.001 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:04.001+0000 7f7fbe968680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-22T13:16:04.012 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:04.012+0000 7f3e4d2b2680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-22T13:16:04.039 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:16:04.039 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-22T13:16:04.070 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:04.069+0000 7f7fbe968680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-22T13:16:04.089 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:04.089+0000 7f3e4d2b2680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-22T13:16:04.106 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:16:04.106 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-09-22T13:16:04.175 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:16:04.175 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-09-22T13:16:04.201 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:16:04.201 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-09-22T13:16:04.268 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:16:04.268 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-09-22T13:16:04.335 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-09-22T13:16:04.335 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-09-22T13:16:04.403 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new a14975d5-629a-4b78-ad9d-a92bed4d99d7 0 2023-09-22T13:16:04.509 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:04.508+0000 7f3e4d2b2680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-22T13:16:04.529 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:04.529+0000 7f7fbe968680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-22T13:16:04.654 INFO:teuthology.orchestra.run.smithi105.stdout:0 2023-09-22T13:16:04.666 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 54717bfc-a890-48fc-92de-5aafba8b0445 1 2023-09-22T13:16:04.876 INFO:teuthology.orchestra.run.smithi105.stdout:1 2023-09-22T13:16:04.888 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 19da5b2b-4afe-4140-9da9-edab7c8b2af5 2 2023-09-22T13:16:05.098 INFO:teuthology.orchestra.run.smithi105.stdout:2 2023-09-22T13:16:05.111 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 813605e5-027d-4227-aea6-80bf3a2e4927 3 2023-09-22T13:16:05.227 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:05.227+0000 7f3e4d2b2680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-22T13:16:05.294 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:05.294+0000 7f3e4d2b2680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-22T13:16:05.321 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:05.320+0000 7f7fbe968680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-22T13:16:05.351 INFO:teuthology.orchestra.run.smithi105.stdout:3 2023-09-22T13:16:05.363 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new 61af8614-ad9a-443a-aa81-163fa5deefaf 4 2023-09-22T13:16:05.393 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:05.392+0000 7f7fbe968680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-22T13:16:05.481 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:05.481+0000 7f3e4d2b2680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-22T13:16:05.571 INFO:teuthology.orchestra.run.smithi105.stdout:4 2023-09-22T13:16:05.583 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new c0c47b6f-ffd0-4d24-b80d-631ed4feaaea 5 2023-09-22T13:16:05.586 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:05.586+0000 7f7fbe968680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-22T13:16:05.792 INFO:teuthology.orchestra.run.smithi105.stdout:5 2023-09-22T13:16:05.806 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new af7d47b9-1402-4843-88cf-13a4a245ffea 6 2023-09-22T13:16:05.919 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:05.919+0000 7f3e4d2b2680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-22T13:16:06.018 INFO:teuthology.orchestra.run.smithi105.stdout:6 2023-09-22T13:16:06.023 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:06.023+0000 7f3e4d2b2680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-22T13:16:06.030 DEBUG:teuthology.orchestra.run.smithi105:> sudo ceph --cluster ceph osd new c7e20682-6337-47f8-9583-5ce29939828d 7 2023-09-22T13:16:06.041 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:06.040+0000 7f7fbe968680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-22T13:16:06.149 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:06.148+0000 7f7fbe968680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-22T13:16:06.175 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:06.175+0000 7f3e4d2b2680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-22T13:16:06.239 INFO:teuthology.orchestra.run.smithi105.stdout:7 2023-09-22T13:16:06.252 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:06.251+0000 7f3e4d2b2680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-22T13:16:06.253 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-22T13:16:06.253 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-22T13:16:06.255 INFO:tasks.ceph.osd.0:Started 2023-09-22T13:16:06.255 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-22T13:16:06.255 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-22T13:16:06.257 INFO:tasks.ceph.osd.1:Started 2023-09-22T13:16:06.257 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-22T13:16:06.257 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-22T13:16:06.259 INFO:tasks.ceph.osd.2:Started 2023-09-22T13:16:06.259 INFO:tasks.ceph.osd.3:Restarting daemon 2023-09-22T13:16:06.259 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-09-22T13:16:06.261 INFO:tasks.ceph.osd.3:Started 2023-09-22T13:16:06.261 INFO:tasks.ceph.osd.4:Restarting daemon 2023-09-22T13:16:06.262 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-09-22T13:16:06.294 INFO:tasks.ceph.osd.4:Started 2023-09-22T13:16:06.295 INFO:tasks.ceph.osd.5:Restarting daemon 2023-09-22T13:16:06.295 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-09-22T13:16:06.297 INFO:tasks.ceph.osd.5:Started 2023-09-22T13:16:06.297 INFO:tasks.ceph.osd.6:Restarting daemon 2023-09-22T13:16:06.297 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-09-22T13:16:06.299 INFO:tasks.ceph.osd.6:Started 2023-09-22T13:16:06.299 INFO:tasks.ceph.osd.7:Restarting daemon 2023-09-22T13:16:06.299 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-09-22T13:16:06.301 INFO:tasks.ceph.osd.7:Started 2023-09-22T13:16:06.301 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-22T13:16:06.309 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:06.309+0000 7f7fbe968680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-22T13:16:06.346 INFO:tasks.ceph.osd.2.smithi096.stderr:2023-09-22T13:16:06.346+0000 7fc943b2a200 -1 Falling back to public interface 2023-09-22T13:16:06.347 INFO:tasks.ceph.osd.0.smithi096.stderr:2023-09-22T13:16:06.347+0000 7f43194b7200 -1 Falling back to public interface 2023-09-22T13:16:06.352 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:06.348+0000 7f3e4d2b2680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-22T13:16:06.363 INFO:tasks.ceph.osd.2.smithi096.stderr:2023-09-22T13:16:06.361+0000 7fc943b2a200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:16:06.363 INFO:tasks.ceph.osd.2.smithi096.stderr:2023-09-22T13:16:06.361+0000 7fc943b2a200 -1 journal do_read_entry(8192): bad header magic 2023-09-22T13:16:06.364 INFO:tasks.ceph.osd.2.smithi096.stderr:2023-09-22T13:16:06.362+0000 7fc943b2a200 -1 journal do_read_entry(8192): bad header magic 2023-09-22T13:16:06.364 INFO:tasks.ceph.osd.0.smithi096.stderr:2023-09-22T13:16:06.364+0000 7f43194b7200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:16:06.365 INFO:tasks.ceph.osd.1.smithi096.stderr:2023-09-22T13:16:06.364+0000 7f26ef4c5200 -1 Falling back to public interface 2023-09-22T13:16:06.370 INFO:tasks.ceph.osd.2.smithi096.stderr:2023-09-22T13:16:06.370+0000 7fc943b2a200 -1 osd.2 0 log_to_monitors {default=true} 2023-09-22T13:16:06.370 INFO:tasks.ceph.osd.0.smithi096.stderr:2023-09-22T13:16:06.364+0000 7f43194b7200 -1 journal do_read_entry(8192): bad header magic 2023-09-22T13:16:06.370 INFO:tasks.ceph.osd.0.smithi096.stderr:2023-09-22T13:16:06.364+0000 7f43194b7200 -1 journal do_read_entry(8192): bad header magic 2023-09-22T13:16:06.373 INFO:tasks.ceph.osd.0.smithi096.stderr:2023-09-22T13:16:06.373+0000 7f43194b7200 -1 osd.0 0 log_to_monitors {default=true} 2023-09-22T13:16:06.382 INFO:tasks.ceph.osd.3.smithi096.stderr:2023-09-22T13:16:06.381+0000 7f056abf7200 -1 Falling back to public interface 2023-09-22T13:16:06.382 INFO:tasks.ceph.osd.5.smithi105.stderr:2023-09-22T13:16:06.382+0000 7f623f56d200 -1 Falling back to public interface 2023-09-22T13:16:06.383 INFO:tasks.ceph.osd.7.smithi105.stderr:2023-09-22T13:16:06.383+0000 7f68c392e200 -1 Falling back to public interface 2023-09-22T13:16:06.387 INFO:tasks.ceph.osd.1.smithi096.stderr:2023-09-22T13:16:06.387+0000 7f26ef4c5200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:16:06.388 INFO:tasks.ceph.osd.1.smithi096.stderr:2023-09-22T13:16:06.387+0000 7f26ef4c5200 -1 journal do_read_entry(8192): bad header magic 2023-09-22T13:16:06.389 INFO:tasks.ceph.osd.1.smithi096.stderr:2023-09-22T13:16:06.387+0000 7f26ef4c5200 -1 journal do_read_entry(8192): bad header magic 2023-09-22T13:16:06.394 INFO:tasks.ceph.osd.3.smithi096.stderr:2023-09-22T13:16:06.394+0000 7f056abf7200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:16:06.394 INFO:tasks.ceph.osd.3.smithi096.stderr:2023-09-22T13:16:06.394+0000 7f056abf7200 -1 journal do_read_entry(8192): bad header magic 2023-09-22T13:16:06.394 INFO:tasks.ceph.osd.3.smithi096.stderr:2023-09-22T13:16:06.394+0000 7f056abf7200 -1 journal do_read_entry(8192): bad header magic 2023-09-22T13:16:06.396 INFO:tasks.ceph.osd.1.smithi096.stderr:2023-09-22T13:16:06.396+0000 7f26ef4c5200 -1 osd.1 0 log_to_monitors {default=true} 2023-09-22T13:16:06.400 INFO:tasks.ceph.osd.6.smithi105.stderr:2023-09-22T13:16:06.399+0000 7f563deba200 -1 Falling back to public interface 2023-09-22T13:16:06.400 INFO:tasks.ceph.osd.5.smithi105.stderr:2023-09-22T13:16:06.400+0000 7f623f56d200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:16:06.401 INFO:tasks.ceph.osd.3.smithi096.stderr:2023-09-22T13:16:06.400+0000 7f056abf7200 -1 osd.3 0 log_to_monitors {default=true} 2023-09-22T13:16:06.401 INFO:tasks.ceph.osd.5.smithi105.stderr:2023-09-22T13:16:06.400+0000 7f623f56d200 -1 journal do_read_entry(8192): bad header magic 2023-09-22T13:16:06.401 INFO:tasks.ceph.osd.5.smithi105.stderr:2023-09-22T13:16:06.400+0000 7f623f56d200 -1 journal do_read_entry(8192): bad header magic 2023-09-22T13:16:06.404 INFO:tasks.ceph.osd.4.smithi105.stderr:2023-09-22T13:16:06.403+0000 7f5efb887200 -1 Falling back to public interface 2023-09-22T13:16:06.404 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:06.404+0000 7f7fbe968680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-22T13:16:06.408 INFO:tasks.ceph.osd.5.smithi105.stderr:2023-09-22T13:16:06.408+0000 7f623f56d200 -1 osd.5 0 log_to_monitors {default=true} 2023-09-22T13:16:06.408 INFO:tasks.ceph.osd.7.smithi105.stderr:2023-09-22T13:16:06.408+0000 7f68c392e200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:16:06.413 INFO:tasks.ceph.osd.7.smithi105.stderr:2023-09-22T13:16:06.409+0000 7f68c392e200 -1 journal do_read_entry(8192): bad header magic 2023-09-22T13:16:06.413 INFO:tasks.ceph.osd.7.smithi105.stderr:2023-09-22T13:16:06.409+0000 7f68c392e200 -1 journal do_read_entry(8192): bad header magic 2023-09-22T13:16:06.415 INFO:tasks.ceph.osd.6.smithi105.stderr:2023-09-22T13:16:06.414+0000 7f563deba200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:16:06.415 INFO:tasks.ceph.osd.6.smithi105.stderr:2023-09-22T13:16:06.415+0000 7f563deba200 -1 journal do_read_entry(8192): bad header magic 2023-09-22T13:16:06.415 INFO:tasks.ceph.osd.6.smithi105.stderr:2023-09-22T13:16:06.415+0000 7f563deba200 -1 journal do_read_entry(8192): bad header magic 2023-09-22T13:16:06.419 INFO:tasks.ceph.osd.7.smithi105.stderr:2023-09-22T13:16:06.418+0000 7f68c392e200 -1 osd.7 0 log_to_monitors {default=true} 2023-09-22T13:16:06.424 INFO:tasks.ceph.osd.6.smithi105.stderr:2023-09-22T13:16:06.423+0000 7f563deba200 -1 osd.6 0 log_to_monitors {default=true} 2023-09-22T13:16:06.424 INFO:tasks.ceph.osd.4.smithi105.stderr:2023-09-22T13:16:06.424+0000 7f5efb887200 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-09-22T13:16:06.425 INFO:tasks.ceph.osd.4.smithi105.stderr:2023-09-22T13:16:06.424+0000 7f5efb887200 -1 journal do_read_entry(8192): bad header magic 2023-09-22T13:16:06.425 INFO:tasks.ceph.osd.4.smithi105.stderr:2023-09-22T13:16:06.424+0000 7f5efb887200 -1 journal do_read_entry(8192): bad header magic 2023-09-22T13:16:06.432 INFO:tasks.ceph.osd.4.smithi105.stderr:2023-09-22T13:16:06.432+0000 7f5efb887200 -1 osd.4 0 log_to_monitors {default=true} 2023-09-22T13:16:06.480 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:06.480+0000 7f7fbe968680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-22T13:16:06.525 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:16:06.525 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":10,"fsid":"37ee0465-b78b-4c40-811f-70b4ab3c2f62","created":"2023-09-22T13:16:03.849297+0000","modified":"2023-09-22T13:16:06.234690+0000","last_up_change":"0.000000","last_in_change":"2023-09-22T13:16:06.234690+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"a14975d5-629a-4b78-ad9d-a92bed4d99d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"54717bfc-a890-48fc-92de-5aafba8b0445","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"19da5b2b-4afe-4140-9da9-edab7c8b2af5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"813605e5-027d-4227-aea6-80bf3a2e4927","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61af8614-ad9a-443a-aa81-163fa5deefaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c0c47b6f-ffd0-4d24-b80d-631ed4feaaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"af7d47b9-1402-4843-88cf-13a4a245ffea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c7e20682-6337-47f8-9583-5ce29939828d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-22T13:16:06.537 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-22T13:16:06.537 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-22T13:16:06.640 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:06.640+0000 7f3e4d2b2680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-22T13:16:06.737 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:06.737+0000 7f7fbe968680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-22T13:16:06.832 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:06.832+0000 7f3e4d2b2680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-22T13:16:06.930 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:06.929+0000 7f7fbe968680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-22T13:16:07.342 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-22T13:16:07.499 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:07.499+0000 7f3e4d2b2680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-22T13:16:07.529 INFO:teuthology.misc.health.smithi096.stdout: 2023-09-22T13:16:07.529 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":11,"fsid":"37ee0465-b78b-4c40-811f-70b4ab3c2f62","created":"2023-09-22T13:16:03.849297+0000","modified":"2023-09-22T13:16:07.235692+0000","last_up_change":"0.000000","last_in_change":"2023-09-22T13:16:06.234690+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"a14975d5-629a-4b78-ad9d-a92bed4d99d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"54717bfc-a890-48fc-92de-5aafba8b0445","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"19da5b2b-4afe-4140-9da9-edab7c8b2af5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"813605e5-027d-4227-aea6-80bf3a2e4927","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61af8614-ad9a-443a-aa81-163fa5deefaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c0c47b6f-ffd0-4d24-b80d-631ed4feaaea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"af7d47b9-1402-4843-88cf-13a4a245ffea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c7e20682-6337-47f8-9583-5ce29939828d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-22T13:16:07.540 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-22T13:16:07.571 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:07.570+0000 7f3e4d2b2680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-22T13:16:07.584 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:07.584+0000 7f7fbe968680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-22T13:16:07.648 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:07.648+0000 7f3e4d2b2680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-22T13:16:07.652 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:07.651+0000 7f7fbe968680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-22T13:16:07.728 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:07.728+0000 7f7fbe968680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-22T13:16:07.795 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:07.795+0000 7f3e4d2b2680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-22T13:16:07.866 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:07.866+0000 7f3e4d2b2680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-22T13:16:07.874 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:07.874+0000 7f7fbe968680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-22T13:16:07.945 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:07.944+0000 7f7fbe968680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-22T13:16:07.980 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:07.980+0000 7f3e4d2b2680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-22T13:16:08.058 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:08.057+0000 7f7fbe968680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-22T13:16:08.088 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:08.087+0000 7f3e4d2b2680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-22T13:16:08.164 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:08.164+0000 7f7fbe968680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-22T13:16:08.239 INFO:tasks.ceph.osd.2.smithi096.stderr:2023-09-22T13:16:08.239+0000 7fc93accc700 -1 osd.2 0 waiting for initial osdmap 2023-09-22T13:16:08.239 INFO:tasks.ceph.osd.3.smithi096.stderr:2023-09-22T13:16:08.239+0000 7f0561d99700 -1 osd.3 0 waiting for initial osdmap 2023-09-22T13:16:08.240 INFO:tasks.ceph.osd.7.smithi105.stderr:2023-09-22T13:16:08.239+0000 7f68baad0700 -1 osd.7 0 waiting for initial osdmap 2023-09-22T13:16:08.240 INFO:tasks.ceph.osd.6.smithi105.stderr:2023-09-22T13:16:08.239+0000 7f563505c700 -1 osd.6 0 waiting for initial osdmap 2023-09-22T13:16:08.240 INFO:tasks.ceph.osd.4.smithi105.stderr:2023-09-22T13:16:08.239+0000 7f5ef2a29700 -1 osd.4 0 waiting for initial osdmap 2023-09-22T13:16:08.241 INFO:tasks.ceph.osd.5.smithi105.stderr:2023-09-22T13:16:08.240+0000 7f623670f700 -1 osd.5 0 waiting for initial osdmap 2023-09-22T13:16:08.241 INFO:tasks.ceph.osd.1.smithi096.stderr:2023-09-22T13:16:08.240+0000 7f26e6667700 -1 osd.1 0 waiting for initial osdmap 2023-09-22T13:16:08.241 INFO:tasks.ceph.osd.0.smithi096.stderr:2023-09-22T13:16:08.240+0000 7f4310659700 -1 osd.0 0 waiting for initial osdmap 2023-09-22T13:16:08.242 INFO:tasks.ceph.osd.3.smithi096.stderr:2023-09-22T13:16:08.241+0000 7f055dd14700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-22T13:16:08.242 INFO:tasks.ceph.osd.7.smithi105.stderr:2023-09-22T13:16:08.241+0000 7f68b6a4b700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-22T13:16:08.243 INFO:tasks.ceph.osd.1.smithi096.stderr:2023-09-22T13:16:08.242+0000 7f26e25e2700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-22T13:16:08.243 INFO:tasks.ceph.osd.6.smithi105.stderr:2023-09-22T13:16:08.243+0000 7f5630fd7700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-22T13:16:08.243 INFO:tasks.ceph.osd.0.smithi096.stderr:2023-09-22T13:16:08.243+0000 7f430c5d4700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-22T13:16:08.244 INFO:tasks.ceph.osd.5.smithi105.stderr:2023-09-22T13:16:08.243+0000 7f623268a700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-22T13:16:08.250 INFO:tasks.ceph.osd.2.smithi096.stderr:2023-09-22T13:16:08.250+0000 7fc936c47700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-22T13:16:08.251 INFO:tasks.ceph.osd.4.smithi105.stderr:2023-09-22T13:16:08.250+0000 7f5eee9a4700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-22T13:16:08.409 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:08.409+0000 7f3e4d2b2680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-22T13:16:08.480 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:08.479+0000 7f3e4d2b2680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-22T13:16:08.484 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:08.484+0000 7f7fbe968680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-22T13:16:08.557 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-22T13:16:08.556+0000 7f7fbe968680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-22T13:16:09.494 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:09.494+0000 7f3e027ad700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-22T13:16:09.495 INFO:tasks.ceph.mgr.y.smithi096.stderr:2023-09-22T13:16:09.495+0000 7f3e027ad700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-22T13:16:14.347 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-22T13:16:14.639 INFO:teuthology.misc.health.smithi096.stdout: 2023-09-22T13:16:14.640 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":16,"fsid":"37ee0465-b78b-4c40-811f-70b4ab3c2f62","created":"2023-09-22T13:16:03.849297+0000","modified":"2023-09-22T13:16:12.264341+0000","last_up_change":"2023-09-22T13:16:09.242072+0000","last_in_change":"2023-09-22T13:16:06.234690+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-09-22T13:16:09.279032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"a14975d5-629a-4b78-ad9d-a92bed4d99d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6807","nonce":140797},{"type":"v1","addr":"172.21.15.96:6809","nonce":140797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":140797},{"type":"v1","addr":"172.21.15.96:6811","nonce":140797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":140797},{"type":"v1","addr":"172.21.15.96:6815","nonce":140797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":140797},{"type":"v1","addr":"172.21.15.96:6813","nonce":140797}]},"public_addr":"172.21.15.96:6809/140797","cluster_addr":"172.21.15.96:6811/140797","heartbeat_back_addr":"172.21.15.96:6815/140797","heartbeat_front_addr":"172.21.15.96:6813/140797","state":["exists","up"]},{"osd":1,"uuid":"54717bfc-a890-48fc-92de-5aafba8b0445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":140810},{"type":"v1","addr":"172.21.15.96:6817","nonce":140810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":140810},{"type":"v1","addr":"172.21.15.96:6819","nonce":140810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":140810},{"type":"v1","addr":"172.21.15.96:6823","nonce":140810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":140810},{"type":"v1","addr":"172.21.15.96:6821","nonce":140810}]},"public_addr":"172.21.15.96:6817/140810","cluster_addr":"172.21.15.96:6819/140810","heartbeat_back_addr":"172.21.15.96:6823/140810","heartbeat_front_addr":"172.21.15.96:6821/140810","state":["exists","up"]},{"osd":2,"uuid":"19da5b2b-4afe-4140-9da9-edab7c8b2af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":140799},{"type":"v1","addr":"172.21.15.96:6801","nonce":140799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":140799},{"type":"v1","addr":"172.21.15.96:6803","nonce":140799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":140799},{"type":"v1","addr":"172.21.15.96:6808","nonce":140799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":140799},{"type":"v1","addr":"172.21.15.96:6805","nonce":140799}]},"public_addr":"172.21.15.96:6801/140799","cluster_addr":"172.21.15.96:6803/140799","heartbeat_back_addr":"172.21.15.96:6808/140799","heartbeat_front_addr":"172.21.15.96:6805/140799","state":["exists","up"]},{"osd":3,"uuid":"813605e5-027d-4227-aea6-80bf3a2e4927","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":140822},{"type":"v1","addr":"172.21.15.96:6825","nonce":140822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":140822},{"type":"v1","addr":"172.21.15.96:6827","nonce":140822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":140822},{"type":"v1","addr":"172.21.15.96:6831","nonce":140822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":140822},{"type":"v1","addr":"172.21.15.96:6829","nonce":140822}]},"public_addr":"172.21.15.96:6825/140822","cluster_addr":"172.21.15.96:6827/140822","heartbeat_back_addr":"172.21.15.96:6831/140822","heartbeat_front_addr":"172.21.15.96:6829/140822","state":["exists","up"]},{"osd":4,"uuid":"61af8614-ad9a-443a-aa81-163fa5deefaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":140707},{"type":"v1","addr":"172.21.15.105:6825","nonce":140707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":140707},{"type":"v1","addr":"172.21.15.105:6827","nonce":140707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":140707},{"type":"v1","addr":"172.21.15.105:6831","nonce":140707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":140707},{"type":"v1","addr":"172.21.15.105:6829","nonce":140707}]},"public_addr":"172.21.15.105:6825/140707","cluster_addr":"172.21.15.105:6827/140707","heartbeat_back_addr":"172.21.15.105:6831/140707","heartbeat_front_addr":"172.21.15.105:6829/140707","state":["exists","up"]},{"osd":5,"uuid":"c0c47b6f-ffd0-4d24-b80d-631ed4feaaea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":140694},{"type":"v1","addr":"172.21.15.105:6801","nonce":140694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":140694},{"type":"v1","addr":"172.21.15.105:6803","nonce":140694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":140694},{"type":"v1","addr":"172.21.15.105:6807","nonce":140694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":140694},{"type":"v1","addr":"172.21.15.105:6805","nonce":140694}]},"public_addr":"172.21.15.105:6801/140694","cluster_addr":"172.21.15.105:6803/140694","heartbeat_back_addr":"172.21.15.105:6807/140694","heartbeat_front_addr":"172.21.15.105:6805/140694","state":["exists","up"]},{"osd":6,"uuid":"af7d47b9-1402-4843-88cf-13a4a245ffea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":140714},{"type":"v1","addr":"172.21.15.105:6817","nonce":140714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":140714},{"type":"v1","addr":"172.21.15.105:6819","nonce":140714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":140714},{"type":"v1","addr":"172.21.15.105:6823","nonce":140714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":140714},{"type":"v1","addr":"172.21.15.105:6821","nonce":140714}]},"public_addr":"172.21.15.105:6817/140714","cluster_addr":"172.21.15.105:6819/140714","heartbeat_back_addr":"172.21.15.105:6823/140714","heartbeat_front_addr":"172.21.15.105:6821/140714","state":["exists","up"]},{"osd":7,"uuid":"c7e20682-6337-47f8-9583-5ce29939828d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":140695},{"type":"v1","addr":"172.21.15.105:6809","nonce":140695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":140695},{"type":"v1","addr":"172.21.15.105:6811","nonce":140695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":140695},{"type":"v1","addr":"172.21.15.105:6815","nonce":140695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":140695},{"type":"v1","addr":"172.21.15.105:6813","nonce":140695}]},"public_addr":"172.21.15.105:6809/140695","cluster_addr":"172.21.15.105:6811/140695","heartbeat_back_addr":"172.21.15.105:6815/140695","heartbeat_front_addr":"172.21.15.105:6813/140695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.421592+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.365312+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.369942+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.366925+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.400652+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.403852+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.451824+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.413643+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-22T13:16:14.651 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-09-22T13:16:14.652 INFO:tasks.ceph:Creating RBD pool 2023-09-22T13:16:14.652 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-22T13:16:15.627 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'rbd' created 2023-09-22T13:16:15.640 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-22T13:16:16.635 INFO:teuthology.orchestra.run.smithi096.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-22T13:16:16.647 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-22T13:16:16.647 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-22T13:16:16.647 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-22T13:16:16.647 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-22T13:16:16.648 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-22T13:16:16.983 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:16:16.995 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":4,"active_gid":4100,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":140497},{"type":"v1","addr":"172.21.15.96:6833","nonce":140497}]},"active_addr":"172.21.15.96:6833/140497","active_change":"2023-09-22T13:16:08.482348+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[{"gid":4105,"name":"x","mgr_features":4540138314316775423,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":2533508159}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":1847137111}]}]} 2023-09-22T13:16:17.000 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-22T13:16:17.000 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-22T13:16:17.000 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-22T13:16:17.290 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:16:17.291 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":18,"fsid":"37ee0465-b78b-4c40-811f-70b4ab3c2f62","created":"2023-09-22T13:16:03.849297+0000","modified":"2023-09-22T13:16:16.620501+0000","last_up_change":"2023-09-22T13:16:09.242072+0000","last_in_change":"2023-09-22T13:16:06.234690+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-09-22T13:16:09.279032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-22T13:16:14.957511+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a14975d5-629a-4b78-ad9d-a92bed4d99d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6807","nonce":140797},{"type":"v1","addr":"172.21.15.96:6809","nonce":140797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":140797},{"type":"v1","addr":"172.21.15.96:6811","nonce":140797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":140797},{"type":"v1","addr":"172.21.15.96:6815","nonce":140797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":140797},{"type":"v1","addr":"172.21.15.96:6813","nonce":140797}]},"public_addr":"172.21.15.96:6809/140797","cluster_addr":"172.21.15.96:6811/140797","heartbeat_back_addr":"172.21.15.96:6815/140797","heartbeat_front_addr":"172.21.15.96:6813/140797","state":["exists","up"]},{"osd":1,"uuid":"54717bfc-a890-48fc-92de-5aafba8b0445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":140810},{"type":"v1","addr":"172.21.15.96:6817","nonce":140810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":140810},{"type":"v1","addr":"172.21.15.96:6819","nonce":140810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":140810},{"type":"v1","addr":"172.21.15.96:6823","nonce":140810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":140810},{"type":"v1","addr":"172.21.15.96:6821","nonce":140810}]},"public_addr":"172.21.15.96:6817/140810","cluster_addr":"172.21.15.96:6819/140810","heartbeat_back_addr":"172.21.15.96:6823/140810","heartbeat_front_addr":"172.21.15.96:6821/140810","state":["exists","up"]},{"osd":2,"uuid":"19da5b2b-4afe-4140-9da9-edab7c8b2af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":140799},{"type":"v1","addr":"172.21.15.96:6801","nonce":140799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":140799},{"type":"v1","addr":"172.21.15.96:6803","nonce":140799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":140799},{"type":"v1","addr":"172.21.15.96:6808","nonce":140799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":140799},{"type":"v1","addr":"172.21.15.96:6805","nonce":140799}]},"public_addr":"172.21.15.96:6801/140799","cluster_addr":"172.21.15.96:6803/140799","heartbeat_back_addr":"172.21.15.96:6808/140799","heartbeat_front_addr":"172.21.15.96:6805/140799","state":["exists","up"]},{"osd":3,"uuid":"813605e5-027d-4227-aea6-80bf3a2e4927","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":140822},{"type":"v1","addr":"172.21.15.96:6825","nonce":140822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":140822},{"type":"v1","addr":"172.21.15.96:6827","nonce":140822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":140822},{"type":"v1","addr":"172.21.15.96:6831","nonce":140822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":140822},{"type":"v1","addr":"172.21.15.96:6829","nonce":140822}]},"public_addr":"172.21.15.96:6825/140822","cluster_addr":"172.21.15.96:6827/140822","heartbeat_back_addr":"172.21.15.96:6831/140822","heartbeat_front_addr":"172.21.15.96:6829/140822","state":["exists","up"]},{"osd":4,"uuid":"61af8614-ad9a-443a-aa81-163fa5deefaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":140707},{"type":"v1","addr":"172.21.15.105:6825","nonce":140707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":140707},{"type":"v1","addr":"172.21.15.105:6827","nonce":140707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":140707},{"type":"v1","addr":"172.21.15.105:6831","nonce":140707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":140707},{"type":"v1","addr":"172.21.15.105:6829","nonce":140707}]},"public_addr":"172.21.15.105:6825/140707","cluster_addr":"172.21.15.105:6827/140707","heartbeat_back_addr":"172.21.15.105:6831/140707","heartbeat_front_addr":"172.21.15.105:6829/140707","state":["exists","up"]},{"osd":5,"uuid":"c0c47b6f-ffd0-4d24-b80d-631ed4feaaea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":140694},{"type":"v1","addr":"172.21.15.105:6801","nonce":140694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":140694},{"type":"v1","addr":"172.21.15.105:6803","nonce":140694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":140694},{"type":"v1","addr":"172.21.15.105:6807","nonce":140694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":140694},{"type":"v1","addr":"172.21.15.105:6805","nonce":140694}]},"public_addr":"172.21.15.105:6801/140694","cluster_addr":"172.21.15.105:6803/140694","heartbeat_back_addr":"172.21.15.105:6807/140694","heartbeat_front_addr":"172.21.15.105:6805/140694","state":["exists","up"]},{"osd":6,"uuid":"af7d47b9-1402-4843-88cf-13a4a245ffea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":140714},{"type":"v1","addr":"172.21.15.105:6817","nonce":140714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":140714},{"type":"v1","addr":"172.21.15.105:6819","nonce":140714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":140714},{"type":"v1","addr":"172.21.15.105:6823","nonce":140714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":140714},{"type":"v1","addr":"172.21.15.105:6821","nonce":140714}]},"public_addr":"172.21.15.105:6817/140714","cluster_addr":"172.21.15.105:6819/140714","heartbeat_back_addr":"172.21.15.105:6823/140714","heartbeat_front_addr":"172.21.15.105:6821/140714","state":["exists","up"]},{"osd":7,"uuid":"c7e20682-6337-47f8-9583-5ce29939828d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":140695},{"type":"v1","addr":"172.21.15.105:6809","nonce":140695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":140695},{"type":"v1","addr":"172.21.15.105:6811","nonce":140695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":140695},{"type":"v1","addr":"172.21.15.105:6815","nonce":140695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":140695},{"type":"v1","addr":"172.21.15.105:6813","nonce":140695}]},"public_addr":"172.21.15.105:6809/140695","cluster_addr":"172.21.15.105:6811/140695","heartbeat_back_addr":"172.21.15.105:6815/140695","heartbeat_front_addr":"172.21.15.105:6813/140695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.421592+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.365312+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.369942+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.366925+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.400652+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.403852+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.451824+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.413643+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-22T13:16:17.304 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-22T13:16:17.304 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-22T13:16:17.590 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:16:17.591 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":18,"fsid":"37ee0465-b78b-4c40-811f-70b4ab3c2f62","created":"2023-09-22T13:16:03.849297+0000","modified":"2023-09-22T13:16:16.620501+0000","last_up_change":"2023-09-22T13:16:09.242072+0000","last_in_change":"2023-09-22T13:16:06.234690+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-09-22T13:16:09.279032+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-22T13:16:14.957511+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a14975d5-629a-4b78-ad9d-a92bed4d99d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6807","nonce":140797},{"type":"v1","addr":"172.21.15.96:6809","nonce":140797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":140797},{"type":"v1","addr":"172.21.15.96:6811","nonce":140797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":140797},{"type":"v1","addr":"172.21.15.96:6815","nonce":140797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":140797},{"type":"v1","addr":"172.21.15.96:6813","nonce":140797}]},"public_addr":"172.21.15.96:6809/140797","cluster_addr":"172.21.15.96:6811/140797","heartbeat_back_addr":"172.21.15.96:6815/140797","heartbeat_front_addr":"172.21.15.96:6813/140797","state":["exists","up"]},{"osd":1,"uuid":"54717bfc-a890-48fc-92de-5aafba8b0445","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":140810},{"type":"v1","addr":"172.21.15.96:6817","nonce":140810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":140810},{"type":"v1","addr":"172.21.15.96:6819","nonce":140810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":140810},{"type":"v1","addr":"172.21.15.96:6823","nonce":140810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":140810},{"type":"v1","addr":"172.21.15.96:6821","nonce":140810}]},"public_addr":"172.21.15.96:6817/140810","cluster_addr":"172.21.15.96:6819/140810","heartbeat_back_addr":"172.21.15.96:6823/140810","heartbeat_front_addr":"172.21.15.96:6821/140810","state":["exists","up"]},{"osd":2,"uuid":"19da5b2b-4afe-4140-9da9-edab7c8b2af5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":140799},{"type":"v1","addr":"172.21.15.96:6801","nonce":140799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":140799},{"type":"v1","addr":"172.21.15.96:6803","nonce":140799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":140799},{"type":"v1","addr":"172.21.15.96:6808","nonce":140799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":140799},{"type":"v1","addr":"172.21.15.96:6805","nonce":140799}]},"public_addr":"172.21.15.96:6801/140799","cluster_addr":"172.21.15.96:6803/140799","heartbeat_back_addr":"172.21.15.96:6808/140799","heartbeat_front_addr":"172.21.15.96:6805/140799","state":["exists","up"]},{"osd":3,"uuid":"813605e5-027d-4227-aea6-80bf3a2e4927","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":140822},{"type":"v1","addr":"172.21.15.96:6825","nonce":140822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":140822},{"type":"v1","addr":"172.21.15.96:6827","nonce":140822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":140822},{"type":"v1","addr":"172.21.15.96:6831","nonce":140822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":140822},{"type":"v1","addr":"172.21.15.96:6829","nonce":140822}]},"public_addr":"172.21.15.96:6825/140822","cluster_addr":"172.21.15.96:6827/140822","heartbeat_back_addr":"172.21.15.96:6831/140822","heartbeat_front_addr":"172.21.15.96:6829/140822","state":["exists","up"]},{"osd":4,"uuid":"61af8614-ad9a-443a-aa81-163fa5deefaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":140707},{"type":"v1","addr":"172.21.15.105:6825","nonce":140707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":140707},{"type":"v1","addr":"172.21.15.105:6827","nonce":140707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":140707},{"type":"v1","addr":"172.21.15.105:6831","nonce":140707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":140707},{"type":"v1","addr":"172.21.15.105:6829","nonce":140707}]},"public_addr":"172.21.15.105:6825/140707","cluster_addr":"172.21.15.105:6827/140707","heartbeat_back_addr":"172.21.15.105:6831/140707","heartbeat_front_addr":"172.21.15.105:6829/140707","state":["exists","up"]},{"osd":5,"uuid":"c0c47b6f-ffd0-4d24-b80d-631ed4feaaea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":140694},{"type":"v1","addr":"172.21.15.105:6801","nonce":140694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":140694},{"type":"v1","addr":"172.21.15.105:6803","nonce":140694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":140694},{"type":"v1","addr":"172.21.15.105:6807","nonce":140694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":140694},{"type":"v1","addr":"172.21.15.105:6805","nonce":140694}]},"public_addr":"172.21.15.105:6801/140694","cluster_addr":"172.21.15.105:6803/140694","heartbeat_back_addr":"172.21.15.105:6807/140694","heartbeat_front_addr":"172.21.15.105:6805/140694","state":["exists","up"]},{"osd":6,"uuid":"af7d47b9-1402-4843-88cf-13a4a245ffea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":140714},{"type":"v1","addr":"172.21.15.105:6817","nonce":140714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":140714},{"type":"v1","addr":"172.21.15.105:6819","nonce":140714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":140714},{"type":"v1","addr":"172.21.15.105:6823","nonce":140714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":140714},{"type":"v1","addr":"172.21.15.105:6821","nonce":140714}]},"public_addr":"172.21.15.105:6817/140714","cluster_addr":"172.21.15.105:6819/140714","heartbeat_back_addr":"172.21.15.105:6823/140714","heartbeat_front_addr":"172.21.15.105:6821/140714","state":["exists","up"]},{"osd":7,"uuid":"c7e20682-6337-47f8-9583-5ce29939828d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":140695},{"type":"v1","addr":"172.21.15.105:6809","nonce":140695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":140695},{"type":"v1","addr":"172.21.15.105:6811","nonce":140695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":140695},{"type":"v1","addr":"172.21.15.105:6815","nonce":140695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":140695},{"type":"v1","addr":"172.21.15.105:6813","nonce":140695}]},"public_addr":"172.21.15.105:6809/140695","cluster_addr":"172.21.15.105:6811/140695","heartbeat_back_addr":"172.21.15.105:6815/140695","heartbeat_front_addr":"172.21.15.105:6813/140695","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.421592+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.365312+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.369942+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.366925+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.400652+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.403852+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.451824+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-09-22T13:16:07.413643+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-22T13:16:17.604 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-22T13:16:17.752 INFO:teuthology.orchestra.run.smithi096.stdout:55834574851 2023-09-22T13:16:17.752 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-22T13:16:17.900 INFO:teuthology.orchestra.run.smithi096.stdout:55834574851 2023-09-22T13:16:17.900 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-22T13:16:18.046 INFO:teuthology.orchestra.run.smithi096.stdout:55834574851 2023-09-22T13:16:18.047 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-09-22T13:16:18.194 INFO:teuthology.orchestra.run.smithi096.stdout:55834574851 2023-09-22T13:16:18.194 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-09-22T13:16:18.343 INFO:teuthology.orchestra.run.smithi096.stdout:55834574851 2023-09-22T13:16:18.344 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-09-22T13:16:18.492 INFO:teuthology.orchestra.run.smithi096.stdout:55834574851 2023-09-22T13:16:18.493 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-09-22T13:16:18.644 INFO:teuthology.orchestra.run.smithi096.stdout:55834574851 2023-09-22T13:16:18.645 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-09-22T13:16:18.794 INFO:teuthology.orchestra.run.smithi096.stdout:55834574851 2023-09-22T13:16:18.794 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-22T13:16:19.084 INFO:teuthology.orchestra.run.smithi096.stdout:55834574851 2023-09-22T13:16:19.097 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.0 2023-09-22T13:16:19.098 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-22T13:16:19.387 INFO:teuthology.orchestra.run.smithi096.stdout:55834574851 2023-09-22T13:16:19.401 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.1 2023-09-22T13:16:19.401 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-22T13:16:19.691 INFO:teuthology.orchestra.run.smithi096.stdout:55834574851 2023-09-22T13:16:19.705 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.2 2023-09-22T13:16:19.705 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-22T13:16:19.995 INFO:teuthology.orchestra.run.smithi096.stdout:55834574851 2023-09-22T13:16:20.008 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.3 2023-09-22T13:16:20.008 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-22T13:16:20.296 INFO:teuthology.orchestra.run.smithi096.stdout:55834574851 2023-09-22T13:16:20.309 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.4 2023-09-22T13:16:20.310 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-22T13:16:20.600 INFO:teuthology.orchestra.run.smithi096.stdout:55834574851 2023-09-22T13:16:20.614 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.5 2023-09-22T13:16:20.614 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-22T13:16:20.905 INFO:teuthology.orchestra.run.smithi096.stdout:55834574852 2023-09-22T13:16:20.918 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574852 for osd.6 2023-09-22T13:16:20.919 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-22T13:16:21.205 INFO:teuthology.orchestra.run.smithi096.stdout:55834574852 2023-09-22T13:16:21.218 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574852 for osd.7 2023-09-22T13:16:21.218 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-22T13:16:21.219 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-22T13:16:21.504 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:16:21.506 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2023-09-22T13:16:21.517 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-09-22T13:16:20.491339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6367364,"kb_used_data":743228028,"kb_used_omap":346,"kb_used_meta":0,"kb_avail":743227676,"statfs":{"total":767585320960,"available":761065140224,"internally_reserved":360448,"allocated":761065500672,"data_stored":761065500672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":355169,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.633605+0000","last_change":"2023-09-22T13:16:16.633440+0000","last_active":"2023-09-22T13:16:16.633605+0000","last_peered":"2023-09-22T13:16:16.633605+0000","last_clean":"2023-09-22T13:16:16.633605+0000","last_became_active":"2023-09-22T13:16:16.633157+0000","last_became_peered":"2023-09-22T13:16:16.633157+0000","last_unstale":"2023-09-22T13:16:16.633605+0000","last_undegraded":"2023-09-22T13:16:16.633605+0000","last_fullsized":"2023-09-22T13:16:16.633605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_clean_scrub_stamp":"2023-09-22T13:16:15.612826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.634935+0000","last_change":"2023-09-22T13:16:16.633832+0000","last_active":"2023-09-22T13:16:16.634935+0000","last_peered":"2023-09-22T13:16:16.634935+0000","last_clean":"2023-09-22T13:16:16.634935+0000","last_became_active":"2023-09-22T13:16:16.633474+0000","last_became_peered":"2023-09-22T13:16:16.633474+0000","last_unstale":"2023-09-22T13:16:16.634935+0000","last_undegraded":"2023-09-22T13:16:16.634935+0000","last_fullsized":"2023-09-22T13:16:16.634935+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_clean_scrub_stamp":"2023-09-22T13:16:15.612826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.778832+0000","last_change":"2023-09-22T13:16:16.778700+0000","last_active":"2023-09-22T13:16:16.778832+0000","last_peered":"2023-09-22T13:16:16.778832+0000","last_clean":"2023-09-22T13:16:16.778832+0000","last_became_active":"2023-09-22T13:16:16.778471+0000","last_became_peered":"2023-09-22T13:16:16.778471+0000","last_unstale":"2023-09-22T13:16:16.778832+0000","last_undegraded":"2023-09-22T13:16:16.778832+0000","last_fullsized":"2023-09-22T13:16:16.778832+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_clean_scrub_stamp":"2023-09-22T13:16:15.612826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.635814+0000","last_change":"2023-09-22T13:16:16.635664+0000","last_active":"2023-09-22T13:16:16.635814+0000","last_peered":"2023-09-22T13:16:16.635814+0000","last_clean":"2023-09-22T13:16:16.635814+0000","last_became_active":"2023-09-22T13:16:16.635362+0000","last_became_peered":"2023-09-22T13:16:16.635362+0000","last_unstale":"2023-09-22T13:16:16.635814+0000","last_undegraded":"2023-09-22T13:16:16.635814+0000","last_fullsized":"2023-09-22T13:16:16.635814+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_clean_scrub_stamp":"2023-09-22T13:16:15.612826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.632227+0000","last_change":"2023-09-22T13:16:11.350375+0000","last_active":"2023-09-22T13:16:16.632227+0000","last_peered":"2023-09-22T13:16:16.632227+0000","last_clean":"2023-09-22T13:16:16.632227+0000","last_became_active":"2023-09-22T13:16:11.350166+0000","last_became_peered":"2023-09-22T13:16:11.350166+0000","last_unstale":"2023-09-22T13:16:16.632227+0000","last_undegraded":"2023-09-22T13:16:16.632227+0000","last_fullsized":"2023-09-22T13:16:16.632227+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:10.253758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:10.253758+0000","last_clean_scrub_stamp":"2023-09-22T13:16:10.253758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.634500+0000","last_change":"2023-09-22T13:16:16.634349+0000","last_active":"2023-09-22T13:16:16.634500+0000","last_peered":"2023-09-22T13:16:16.634500+0000","last_clean":"2023-09-22T13:16:16.634500+0000","last_became_active":"2023-09-22T13:16:16.634033+0000","last_became_peered":"2023-09-22T13:16:16.634033+0000","last_unstale":"2023-09-22T13:16:16.634500+0000","last_undegraded":"2023-09-22T13:16:16.634500+0000","last_fullsized":"2023-09-22T13:16:16.634500+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_clean_scrub_stamp":"2023-09-22T13:16:15.612826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.634569+0000","last_change":"2023-09-22T13:16:16.634402+0000","last_active":"2023-09-22T13:16:16.634569+0000","last_peered":"2023-09-22T13:16:16.634569+0000","last_clean":"2023-09-22T13:16:16.634569+0000","last_became_active":"2023-09-22T13:16:16.634073+0000","last_became_peered":"2023-09-22T13:16:16.634073+0000","last_unstale":"2023-09-22T13:16:16.634569+0000","last_undegraded":"2023-09-22T13:16:16.634569+0000","last_fullsized":"2023-09-22T13:16:16.634569+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_clean_scrub_stamp":"2023-09-22T13:16:15.612826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.635805+0000","last_change":"2023-09-22T13:16:16.635656+0000","last_active":"2023-09-22T13:16:16.635805+0000","last_peered":"2023-09-22T13:16:16.635805+0000","last_clean":"2023-09-22T13:16:16.635805+0000","last_became_active":"2023-09-22T13:16:16.635378+0000","last_became_peered":"2023-09-22T13:16:16.635378+0000","last_unstale":"2023-09-22T13:16:16.635805+0000","last_undegraded":"2023-09-22T13:16:16.635805+0000","last_fullsized":"2023-09-22T13:16:16.635805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_clean_scrub_stamp":"2023-09-22T13:16:15.612826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.778788+0000","last_change":"2023-09-22T13:16:16.778641+0000","last_active":"2023-09-22T13:16:16.778788+0000","last_peered":"2023-09-22T13:16:16.778788+0000","last_clean":"2023-09-22T13:16:16.778788+0000","last_became_active":"2023-09-22T13:16:16.778421+0000","last_became_peered":"2023-09-22T13:16:16.778421+0000","last_unstale":"2023-09-22T13:16:16.778788+0000","last_undegraded":"2023-09-22T13:16:16.778788+0000","last_fullsized":"2023-09-22T13:16:16.778788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_clean_scrub_stamp":"2023-09-22T13:16:15.612826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795952,"kb_used_data":92903484,"kb_used_omap":53,"kb_used_meta":0,"kb_avail":92903428,"statfs":{"total":95948165120,"available":95133110272,"internally_reserved":57344,"allocated":95133167616,"data_stored":95133167616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54452,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795936,"kb_used_data":92903488,"kb_used_omap":43,"kb_used_meta":0,"kb_avail":92903444,"statfs":{"total":95948165120,"available":95133126656,"internally_reserved":45056,"allocated":95133171712,"data_stored":95133171712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44226,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795968,"kb_used_data":92903484,"kb_used_omap":48,"kb_used_meta":0,"kb_avail":92903412,"statfs":{"total":95948165120,"available":95133093888,"internally_reserved":73728,"allocated":95133167616,"data_stored":95133167616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":49931,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795936,"kb_used_data":92903492,"kb_used_omap":48,"kb_used_meta":0,"kb_avail":92903444,"statfs":{"total":95948165120,"available":95133126656,"internally_reserved":49152,"allocated":95133175808,"data_stored":95133175808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":49270,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795936,"kb_used_data":92903488,"kb_used_omap":43,"kb_used_meta":0,"kb_avail":92903444,"statfs":{"total":95948165120,"available":95133126656,"internally_reserved":45056,"allocated":95133171712,"data_stored":95133171712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44264,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795924,"kb_used_data":92903488,"kb_used_omap":38,"kb_used_meta":0,"kb_avail":92903456,"statfs":{"total":95948165120,"available":95133138944,"internally_reserved":32768,"allocated":95133171712,"data_stored":95133171712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39170,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795780,"kb_used_data":92903616,"kb_used_omap":28,"kb_used_meta":0,"kb_avail":92903600,"statfs":{"total":95948165120,"available":95133286400,"internally_reserved":16384,"allocated":95133302784,"data_stored":95133302784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29624,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795932,"kb_used_data":92903488,"kb_used_omap":43,"kb_used_meta":0,"kb_avail":92903448,"statfs":{"total":95948165120,"available":95133130752,"internally_reserved":40960,"allocated":95133171712,"data_stored":95133171712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44232,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]}]}],"pool_statfs":[]}} 2023-09-22T13:16:21.519 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-22T13:16:21.803 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:16:21.804 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2023-09-22T13:16:21.815 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-09-22T13:16:20.491339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6367364,"kb_used_data":743228028,"kb_used_omap":346,"kb_used_meta":0,"kb_avail":743227676,"statfs":{"total":767585320960,"available":761065140224,"internally_reserved":360448,"allocated":761065500672,"data_stored":761065500672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":355169,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.633605+0000","last_change":"2023-09-22T13:16:16.633440+0000","last_active":"2023-09-22T13:16:16.633605+0000","last_peered":"2023-09-22T13:16:16.633605+0000","last_clean":"2023-09-22T13:16:16.633605+0000","last_became_active":"2023-09-22T13:16:16.633157+0000","last_became_peered":"2023-09-22T13:16:16.633157+0000","last_unstale":"2023-09-22T13:16:16.633605+0000","last_undegraded":"2023-09-22T13:16:16.633605+0000","last_fullsized":"2023-09-22T13:16:16.633605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_clean_scrub_stamp":"2023-09-22T13:16:15.612826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.634935+0000","last_change":"2023-09-22T13:16:16.633832+0000","last_active":"2023-09-22T13:16:16.634935+0000","last_peered":"2023-09-22T13:16:16.634935+0000","last_clean":"2023-09-22T13:16:16.634935+0000","last_became_active":"2023-09-22T13:16:16.633474+0000","last_became_peered":"2023-09-22T13:16:16.633474+0000","last_unstale":"2023-09-22T13:16:16.634935+0000","last_undegraded":"2023-09-22T13:16:16.634935+0000","last_fullsized":"2023-09-22T13:16:16.634935+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_clean_scrub_stamp":"2023-09-22T13:16:15.612826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.778832+0000","last_change":"2023-09-22T13:16:16.778700+0000","last_active":"2023-09-22T13:16:16.778832+0000","last_peered":"2023-09-22T13:16:16.778832+0000","last_clean":"2023-09-22T13:16:16.778832+0000","last_became_active":"2023-09-22T13:16:16.778471+0000","last_became_peered":"2023-09-22T13:16:16.778471+0000","last_unstale":"2023-09-22T13:16:16.778832+0000","last_undegraded":"2023-09-22T13:16:16.778832+0000","last_fullsized":"2023-09-22T13:16:16.778832+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_clean_scrub_stamp":"2023-09-22T13:16:15.612826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.635814+0000","last_change":"2023-09-22T13:16:16.635664+0000","last_active":"2023-09-22T13:16:16.635814+0000","last_peered":"2023-09-22T13:16:16.635814+0000","last_clean":"2023-09-22T13:16:16.635814+0000","last_became_active":"2023-09-22T13:16:16.635362+0000","last_became_peered":"2023-09-22T13:16:16.635362+0000","last_unstale":"2023-09-22T13:16:16.635814+0000","last_undegraded":"2023-09-22T13:16:16.635814+0000","last_fullsized":"2023-09-22T13:16:16.635814+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_clean_scrub_stamp":"2023-09-22T13:16:15.612826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.632227+0000","last_change":"2023-09-22T13:16:11.350375+0000","last_active":"2023-09-22T13:16:16.632227+0000","last_peered":"2023-09-22T13:16:16.632227+0000","last_clean":"2023-09-22T13:16:16.632227+0000","last_became_active":"2023-09-22T13:16:11.350166+0000","last_became_peered":"2023-09-22T13:16:11.350166+0000","last_unstale":"2023-09-22T13:16:16.632227+0000","last_undegraded":"2023-09-22T13:16:16.632227+0000","last_fullsized":"2023-09-22T13:16:16.632227+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:10.253758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:10.253758+0000","last_clean_scrub_stamp":"2023-09-22T13:16:10.253758+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.634500+0000","last_change":"2023-09-22T13:16:16.634349+0000","last_active":"2023-09-22T13:16:16.634500+0000","last_peered":"2023-09-22T13:16:16.634500+0000","last_clean":"2023-09-22T13:16:16.634500+0000","last_became_active":"2023-09-22T13:16:16.634033+0000","last_became_peered":"2023-09-22T13:16:16.634033+0000","last_unstale":"2023-09-22T13:16:16.634500+0000","last_undegraded":"2023-09-22T13:16:16.634500+0000","last_fullsized":"2023-09-22T13:16:16.634500+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_clean_scrub_stamp":"2023-09-22T13:16:15.612826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.634569+0000","last_change":"2023-09-22T13:16:16.634402+0000","last_active":"2023-09-22T13:16:16.634569+0000","last_peered":"2023-09-22T13:16:16.634569+0000","last_clean":"2023-09-22T13:16:16.634569+0000","last_became_active":"2023-09-22T13:16:16.634073+0000","last_became_peered":"2023-09-22T13:16:16.634073+0000","last_unstale":"2023-09-22T13:16:16.634569+0000","last_undegraded":"2023-09-22T13:16:16.634569+0000","last_fullsized":"2023-09-22T13:16:16.634569+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_clean_scrub_stamp":"2023-09-22T13:16:15.612826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.635805+0000","last_change":"2023-09-22T13:16:16.635656+0000","last_active":"2023-09-22T13:16:16.635805+0000","last_peered":"2023-09-22T13:16:16.635805+0000","last_clean":"2023-09-22T13:16:16.635805+0000","last_became_active":"2023-09-22T13:16:16.635378+0000","last_became_peered":"2023-09-22T13:16:16.635378+0000","last_unstale":"2023-09-22T13:16:16.635805+0000","last_undegraded":"2023-09-22T13:16:16.635805+0000","last_fullsized":"2023-09-22T13:16:16.635805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_clean_scrub_stamp":"2023-09-22T13:16:15.612826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-09-22T13:16:16.778788+0000","last_change":"2023-09-22T13:16:16.778641+0000","last_active":"2023-09-22T13:16:16.778788+0000","last_peered":"2023-09-22T13:16:16.778788+0000","last_clean":"2023-09-22T13:16:16.778788+0000","last_became_active":"2023-09-22T13:16:16.778421+0000","last_became_peered":"2023-09-22T13:16:16.778421+0000","last_unstale":"2023-09-22T13:16:16.778788+0000","last_undegraded":"2023-09-22T13:16:16.778788+0000","last_fullsized":"2023-09-22T13:16:16.778788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-22T13:16:15.612826+0000","last_clean_scrub_stamp":"2023-09-22T13:16:15.612826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795952,"kb_used_data":92903484,"kb_used_omap":53,"kb_used_meta":0,"kb_avail":92903428,"statfs":{"total":95948165120,"available":95133110272,"internally_reserved":57344,"allocated":95133167616,"data_stored":95133167616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54452,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795936,"kb_used_data":92903488,"kb_used_omap":43,"kb_used_meta":0,"kb_avail":92903444,"statfs":{"total":95948165120,"available":95133126656,"internally_reserved":45056,"allocated":95133171712,"data_stored":95133171712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44226,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795968,"kb_used_data":92903484,"kb_used_omap":48,"kb_used_meta":0,"kb_avail":92903412,"statfs":{"total":95948165120,"available":95133093888,"internally_reserved":73728,"allocated":95133167616,"data_stored":95133167616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":49931,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795936,"kb_used_data":92903492,"kb_used_omap":48,"kb_used_meta":0,"kb_avail":92903444,"statfs":{"total":95948165120,"available":95133126656,"internally_reserved":49152,"allocated":95133175808,"data_stored":95133175808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":49270,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795936,"kb_used_data":92903488,"kb_used_omap":43,"kb_used_meta":0,"kb_avail":92903444,"statfs":{"total":95948165120,"available":95133126656,"internally_reserved":45056,"allocated":95133171712,"data_stored":95133171712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44264,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795924,"kb_used_data":92903488,"kb_used_omap":38,"kb_used_meta":0,"kb_avail":92903456,"statfs":{"total":95948165120,"available":95133138944,"internally_reserved":32768,"allocated":95133171712,"data_stored":95133171712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39170,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795780,"kb_used_data":92903616,"kb_used_omap":28,"kb_used_meta":0,"kb_avail":92903600,"statfs":{"total":95948165120,"available":95133286400,"internally_reserved":16384,"allocated":95133302784,"data_stored":95133302784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29624,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795932,"kb_used_data":92903488,"kb_used_omap":43,"kb_used_meta":0,"kb_avail":92903448,"statfs":{"total":95948165120,"available":95133130752,"internally_reserved":40960,"allocated":95133171712,"data_stored":95133171712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44232,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]}]}],"pool_statfs":[]}} 2023-09-22T13:16:21.816 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-22T13:16:21.817 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-22T13:16:21.817 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-22T13:16:21.817 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-22T13:16:22.140 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:16:22.152 INFO:teuthology.orchestra.run.smithi096.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-09-22T13:16:22.153 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-09-22T13:16:22.153 INFO:teuthology.run_tasks:Running task openssl_keys... 2023-09-22T13:16:22.165 INFO:teuthology.run_tasks:Running task rgw... 2023-09-22T13:16:22.185 DEBUG:tasks.rgw:config is {'client.0': None} 2023-09-22T13:16:22.185 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2023-09-22T13:16:22.185 INFO:tasks.rgw:Creating data pools 2023-09-22T13:16:22.186 DEBUG:tasks.rgw:Obtaining remote for client client.0 2023-09-22T13:16:22.186 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster ceph 2023-09-22T13:16:22.661 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'default.rgw.buckets.data' created 2023-09-22T13:16:22.672 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2023-09-22T13:16:23.676 INFO:teuthology.orchestra.run.smithi096.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-09-22T13:16:23.688 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2023-09-22T13:16:24.675 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'default.rgw.buckets.index' created 2023-09-22T13:16:24.687 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2023-09-22T13:16:25.684 INFO:teuthology.orchestra.run.smithi096.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-09-22T13:16:25.699 DEBUG:tasks.rgw:Pools created 2023-09-22T13:16:25.699 INFO:tasks.rgw:Configuring compression type = random 2023-09-22T13:16:25.699 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2023-09-22T13:16:25.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2023-09-22T13:16:25.699 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2023-09-22T13:16:25.818 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:16:25.818 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:16:25.825 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:25.825+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.682 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.682+0000 7fb994a3f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:27.682 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.682+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.683 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.683+0000 7fb994a3f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:27.683 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.683+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.683 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.683+0000 7fb994a3f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:27.683 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.683+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.684 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.684+0000 7fb994a3f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:27.713 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.713+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.714 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.713+0000 7fb994a3f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:27.714 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.713+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.714 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.714+0000 7fb994a3f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:27.714 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.714+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.714 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.714+0000 7fb994a3f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:27.714 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.714+0000 7fb994a3f500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:16:27.714 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.714+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.715 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.714+0000 7fb994a3f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:27.715 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.714+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.715 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.714+0000 7fb994a3f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:27.715 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.714+0000 7fb994a3f500 10 Creating default zonegroup 2023-09-22T13:16:27.715 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.714+0000 7fb994a3f500 20 get_system_obj_state: rctx=0x7ffc34f5f600 obj=:.pools.avail state=0x5563e30c9810 s->prefetch_data=0 2023-09-22T13:16:27.715 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.715+0000 7fb994a3f500 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-09-22T13:16:27.720 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.720+0000 7fb980722700 0 -- 172.21.15.96:0/3389047717 >> [v2:172.21.15.105:6808/140695,v1:172.21.15.105:6809/140695] conn(0x7fb97005d070 msgr2=0x7fb97004cac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:16:27.724 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.723+0000 7fb994a3f500 10 WARNING: store->list_zones() returned r=-2 2023-09-22T13:16:27.724 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.723+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.724 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.724+0000 7fb994a3f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:27.725 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.725+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.726+0000 7fb994a3f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.726+0000 7fb994a3f500 10 could not read realm id: (2) No such file or directory 2023-09-22T13:16:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.726+0000 7fb994a3f500 10 WARNING: failed to set zone as default, r=-22 2023-09-22T13:16:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.726+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.726 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.726+0000 7fb994a3f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:27.727 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.727+0000 7fb97f720700 0 -- 172.21.15.96:0/3389047717 >> [v2:172.21.15.105:6800/140694,v1:172.21.15.105:6801/140694] conn(0x5563e320d910 msgr2=0x5563e320fd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:16:27.728 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.728+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.729 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.728+0000 7fb994a3f500 20 rados_obj.operate() r=0 bl.length=873 2023-09-22T13:16:27.729 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.729+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.729 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.729+0000 7fb994a3f500 20 rados_obj.operate() r=0 bl.length=358 2023-09-22T13:16:27.729 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.729+0000 7fb994a3f500 20 zonegroup default 2023-09-22T13:16:27.730 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.729+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.730 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.729+0000 7fb994a3f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:27.730 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.729+0000 7fb994a3f500 10 Cannot find current period zone using local zone 2023-09-22T13:16:27.730 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.729+0000 7fb994a3f500 10 Using default name default 2023-09-22T13:16:27.730 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.729+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.730 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.729+0000 7fb994a3f500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:16:27.730 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.729+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:27.731 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.730+0000 7fb994a3f500 20 rados_obj.operate() r=0 bl.length=873 2023-09-22T13:16:27.731 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.730+0000 7fb994a3f500 20 zone default found 2023-09-22T13:16:27.731 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:27.730+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:29.692 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:29.692+0000 7fb994a3f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:29.692 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:29.692+0000 7fb994a3f500 20 rados->read ofs=0 len=0 2023-09-22T13:16:29.693 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:29.692+0000 7fb994a3f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:29.693 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:29.692+0000 7fb994a3f500 20 started sync module instance, tier type = 2023-09-22T13:16:29.693 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:29.692+0000 7fb994a3f500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:16:31.739 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.738+0000 7fb994a3f500 20 add_watcher() i=0 2023-09-22T13:16:31.739 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.738+0000 7fb994a3f500 20 add_watcher() i=1 2023-09-22T13:16:31.739 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.738+0000 7fb994a3f500 20 add_watcher() i=2 2023-09-22T13:16:31.739 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.738+0000 7fb994a3f500 20 add_watcher() i=3 2023-09-22T13:16:31.739 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.738+0000 7fb994a3f500 20 add_watcher() i=4 2023-09-22T13:16:31.740 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.738+0000 7fb994a3f500 20 add_watcher() i=5 2023-09-22T13:16:31.740 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.738+0000 7fb994a3f500 20 add_watcher() i=6 2023-09-22T13:16:31.741 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.739+0000 7fb994a3f500 20 add_watcher() i=7 2023-09-22T13:16:31.741 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.739+0000 7fb994a3f500 2 all 8 watchers are set, enabling cache 2023-09-22T13:16:31.741 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.740+0000 7fb994a3f500 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):409: oid=data_loggenerations_metadata not found 2023-09-22T13:16:31.741 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.740+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.0 2023-09-22T13:16:31.741 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.740+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.741 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.740+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.0 does not exist 2023-09-22T13:16:31.741 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.740+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.1 2023-09-22T13:16:31.741 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.740+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.741 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.740+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.1 does not exist 2023-09-22T13:16:31.742 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.740+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.2 2023-09-22T13:16:31.742 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.740+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.742 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.741+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.2 does not exist 2023-09-22T13:16:31.742 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.741+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.3 2023-09-22T13:16:31.742 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.741+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.743 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.741+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.3 does not exist 2023-09-22T13:16:31.744 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.741+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.4 2023-09-22T13:16:31.744 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.741+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.744 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.741+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.4 does not exist 2023-09-22T13:16:31.744 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.741+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.5 2023-09-22T13:16:31.744 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.741+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.744 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.741+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.5 does not exist 2023-09-22T13:16:31.744 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.741+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.6 2023-09-22T13:16:31.744 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.741+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.744 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.742+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.6 does not exist 2023-09-22T13:16:31.744 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.742+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.7 2023-09-22T13:16:31.745 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.742+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.745 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.742+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.7 does not exist 2023-09-22T13:16:31.745 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.742+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.8 2023-09-22T13:16:31.745 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.742+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.745 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.742+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.8 does not exist 2023-09-22T13:16:31.745 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.742+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.9 2023-09-22T13:16:31.745 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.742+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.745 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.742+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.9 does not exist 2023-09-22T13:16:31.745 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.742+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.10 2023-09-22T13:16:31.745 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.742+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.745 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.743+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.10 does not exist 2023-09-22T13:16:31.745 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.743+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.11 2023-09-22T13:16:31.746 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.743+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.746 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.743+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.11 does not exist 2023-09-22T13:16:31.746 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.743+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.12 2023-09-22T13:16:31.746 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.743+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.746 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.743+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.12 does not exist 2023-09-22T13:16:31.746 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.743+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.13 2023-09-22T13:16:31.746 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.743+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.746 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.743+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.13 does not exist 2023-09-22T13:16:31.746 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.743+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.14 2023-09-22T13:16:31.746 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.743+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.749 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.744+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.14 does not exist 2023-09-22T13:16:31.749 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.744+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.15 2023-09-22T13:16:31.749 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.744+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.750 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.744+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.15 does not exist 2023-09-22T13:16:31.750 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.744+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.16 2023-09-22T13:16:31.750 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.744+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.750 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.744+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.16 does not exist 2023-09-22T13:16:31.750 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.744+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.17 2023-09-22T13:16:31.750 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.744+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.750 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.745+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.17 does not exist 2023-09-22T13:16:31.750 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.745+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.18 2023-09-22T13:16:31.750 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.745+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.751 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.745+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.18 does not exist 2023-09-22T13:16:31.751 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.745+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.19 2023-09-22T13:16:31.751 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.745+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.751 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.745+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.19 does not exist 2023-09-22T13:16:31.751 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.745+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.20 2023-09-22T13:16:31.751 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.745+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.751 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.746+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.20 does not exist 2023-09-22T13:16:31.751 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.746+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.21 2023-09-22T13:16:31.751 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.746+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.751 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.746+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.21 does not exist 2023-09-22T13:16:31.751 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.746+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.22 2023-09-22T13:16:31.752 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.746+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.752 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.746+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.22 does not exist 2023-09-22T13:16:31.752 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.746+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.23 2023-09-22T13:16:31.752 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.746+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.752 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.746+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.23 does not exist 2023-09-22T13:16:31.752 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.746+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.24 2023-09-22T13:16:31.752 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.746+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.752 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.747+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.24 does not exist 2023-09-22T13:16:31.752 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.747+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.25 2023-09-22T13:16:31.752 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.747+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.752 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.747+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.25 does not exist 2023-09-22T13:16:31.753 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.747+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.26 2023-09-22T13:16:31.753 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.747+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.753 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.747+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.26 does not exist 2023-09-22T13:16:31.753 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.747+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.27 2023-09-22T13:16:31.753 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.747+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.753 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.747+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.27 does not exist 2023-09-22T13:16:31.753 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.747+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.28 2023-09-22T13:16:31.753 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.747+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.753 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.748+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.28 does not exist 2023-09-22T13:16:31.753 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.748+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.29 2023-09-22T13:16:31.754 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.748+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.754 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.748+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.29 does not exist 2023-09-22T13:16:31.754 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.748+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.30 2023-09-22T13:16:31.754 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.748+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.754 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.748+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.30 does not exist 2023-09-22T13:16:31.754 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.748+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.31 2023-09-22T13:16:31.754 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.748+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.754 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.748+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.31 does not exist 2023-09-22T13:16:31.754 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.748+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.32 2023-09-22T13:16:31.754 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.748+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.755 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.749+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.32 does not exist 2023-09-22T13:16:31.755 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.749+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.33 2023-09-22T13:16:31.755 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.749+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.755 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.749+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.33 does not exist 2023-09-22T13:16:31.755 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.749+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.34 2023-09-22T13:16:31.755 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.749+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.755 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.749+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.34 does not exist 2023-09-22T13:16:31.755 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.749+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.35 2023-09-22T13:16:31.755 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.749+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.755 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.749+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.35 does not exist 2023-09-22T13:16:31.755 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.749+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.36 2023-09-22T13:16:31.755 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.749+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.761 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.750+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.36 does not exist 2023-09-22T13:16:31.761 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.750+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.37 2023-09-22T13:16:31.761 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.750+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.761 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.750+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.37 does not exist 2023-09-22T13:16:31.762 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.750+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.38 2023-09-22T13:16:31.762 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.750+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.762 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.750+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.38 does not exist 2023-09-22T13:16:31.762 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.750+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.39 2023-09-22T13:16:31.762 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.750+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.762 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.750+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.39 does not exist 2023-09-22T13:16:31.762 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.750+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.40 2023-09-22T13:16:31.762 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.750+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.762 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.751+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.40 does not exist 2023-09-22T13:16:31.762 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.751+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.41 2023-09-22T13:16:31.762 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.751+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.751+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.41 does not exist 2023-09-22T13:16:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.751+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.42 2023-09-22T13:16:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.751+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.751+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.42 does not exist 2023-09-22T13:16:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.751+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.43 2023-09-22T13:16:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.751+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.751+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.43 does not exist 2023-09-22T13:16:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.751+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.44 2023-09-22T13:16:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.751+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.752+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.44 does not exist 2023-09-22T13:16:31.763 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.752+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.45 2023-09-22T13:16:31.764 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.752+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.764 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.752+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.45 does not exist 2023-09-22T13:16:31.764 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.752+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.46 2023-09-22T13:16:31.764 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.752+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.764 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.752+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.46 does not exist 2023-09-22T13:16:31.764 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.752+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.47 2023-09-22T13:16:31.764 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.752+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.764 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.752+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.47 does not exist 2023-09-22T13:16:31.764 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.752+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.48 2023-09-22T13:16:31.764 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.752+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.764 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.753+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.48 does not exist 2023-09-22T13:16:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.753+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.49 2023-09-22T13:16:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.753+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.753+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.49 does not exist 2023-09-22T13:16:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.753+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.50 2023-09-22T13:16:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.753+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.753+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.50 does not exist 2023-09-22T13:16:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.753+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.51 2023-09-22T13:16:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.753+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.753+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.51 does not exist 2023-09-22T13:16:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.753+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.52 2023-09-22T13:16:31.765 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.753+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.766 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.754+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.52 does not exist 2023-09-22T13:16:31.766 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.754+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.53 2023-09-22T13:16:31.766 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.754+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.766 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.754+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.53 does not exist 2023-09-22T13:16:31.766 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.754+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.54 2023-09-22T13:16:31.766 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.754+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.766 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.754+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.54 does not exist 2023-09-22T13:16:31.766 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.754+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.55 2023-09-22T13:16:31.766 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.754+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.766 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.754+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.55 does not exist 2023-09-22T13:16:31.767 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.754+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.56 2023-09-22T13:16:31.767 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.754+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.767 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.754+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.56 does not exist 2023-09-22T13:16:31.767 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.754+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.57 2023-09-22T13:16:31.767 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.754+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.767 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.755+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.57 does not exist 2023-09-22T13:16:31.767 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.755+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.58 2023-09-22T13:16:31.767 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.755+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.767 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.755+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.58 does not exist 2023-09-22T13:16:31.767 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.755+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.59 2023-09-22T13:16:31.767 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.755+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.768 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.755+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.59 does not exist 2023-09-22T13:16:31.768 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.755+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.60 2023-09-22T13:16:31.768 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.755+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.768 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.755+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.60 does not exist 2023-09-22T13:16:31.768 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.755+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.61 2023-09-22T13:16:31.768 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.755+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.768 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.756+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.61 does not exist 2023-09-22T13:16:31.768 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.756+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.62 2023-09-22T13:16:31.768 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.756+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.768 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.756+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.62 does not exist 2023-09-22T13:16:31.768 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.756+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.63 2023-09-22T13:16:31.768 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.756+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.769 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.756+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.63 does not exist 2023-09-22T13:16:31.769 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.756+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.64 2023-09-22T13:16:31.769 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.756+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.769 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.756+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.64 does not exist 2023-09-22T13:16:31.769 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.756+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.65 2023-09-22T13:16:31.769 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.756+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.769 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.756+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.65 does not exist 2023-09-22T13:16:31.769 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.756+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.66 2023-09-22T13:16:31.769 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.756+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.769 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.757+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.66 does not exist 2023-09-22T13:16:31.769 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.757+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.67 2023-09-22T13:16:31.770 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.757+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.770 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.757+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.67 does not exist 2023-09-22T13:16:31.770 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.757+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.68 2023-09-22T13:16:31.770 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.757+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.770 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.757+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.68 does not exist 2023-09-22T13:16:31.770 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.757+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.69 2023-09-22T13:16:31.770 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.757+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.770 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.758+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.69 does not exist 2023-09-22T13:16:31.770 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.758+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.70 2023-09-22T13:16:31.770 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.758+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.771 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.758+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.70 does not exist 2023-09-22T13:16:31.771 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.758+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.71 2023-09-22T13:16:31.771 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.758+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.771 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.758+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.71 does not exist 2023-09-22T13:16:31.771 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.758+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.72 2023-09-22T13:16:31.771 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.758+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.771 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.758+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.72 does not exist 2023-09-22T13:16:31.771 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.758+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.73 2023-09-22T13:16:31.771 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.758+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.771 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.758+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.73 does not exist 2023-09-22T13:16:31.771 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.758+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.74 2023-09-22T13:16:31.772 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.758+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.772 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.759+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.74 does not exist 2023-09-22T13:16:31.772 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.759+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.75 2023-09-22T13:16:31.772 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.759+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.772 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.759+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.75 does not exist 2023-09-22T13:16:31.772 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.759+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.76 2023-09-22T13:16:31.772 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.759+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.772 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.759+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.76 does not exist 2023-09-22T13:16:31.772 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.759+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.77 2023-09-22T13:16:31.772 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.759+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.772 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.759+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.77 does not exist 2023-09-22T13:16:31.773 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.759+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.78 2023-09-22T13:16:31.773 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.759+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.773 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.760+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.78 does not exist 2023-09-22T13:16:31.773 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.760+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.79 2023-09-22T13:16:31.773 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.760+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.773 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.760+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.79 does not exist 2023-09-22T13:16:31.773 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.760+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.80 2023-09-22T13:16:31.773 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.760+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.773 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.760+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.80 does not exist 2023-09-22T13:16:31.773 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.760+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.81 2023-09-22T13:16:31.773 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.760+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.774 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.760+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.81 does not exist 2023-09-22T13:16:31.774 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.760+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.82 2023-09-22T13:16:31.774 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.760+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.774 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.761+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.82 does not exist 2023-09-22T13:16:31.774 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.761+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.83 2023-09-22T13:16:31.774 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.761+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.774 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.761+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.83 does not exist 2023-09-22T13:16:31.774 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.761+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.84 2023-09-22T13:16:31.774 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.761+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.774 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.761+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.84 does not exist 2023-09-22T13:16:31.775 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.761+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.85 2023-09-22T13:16:31.775 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.761+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.780 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.761+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.85 does not exist 2023-09-22T13:16:31.780 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.761+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.86 2023-09-22T13:16:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.761+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.762+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.86 does not exist 2023-09-22T13:16:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.762+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.87 2023-09-22T13:16:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.762+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.762+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.87 does not exist 2023-09-22T13:16:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.762+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.88 2023-09-22T13:16:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.762+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.762+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.88 does not exist 2023-09-22T13:16:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.762+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.89 2023-09-22T13:16:31.781 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.762+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.762+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.89 does not exist 2023-09-22T13:16:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.762+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.90 2023-09-22T13:16:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.762+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.763+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.90 does not exist 2023-09-22T13:16:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.763+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.91 2023-09-22T13:16:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.763+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.763+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.91 does not exist 2023-09-22T13:16:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.763+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.92 2023-09-22T13:16:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.763+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.782 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.763+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.92 does not exist 2023-09-22T13:16:31.783 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.763+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.93 2023-09-22T13:16:31.783 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.763+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.783 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.763+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.93 does not exist 2023-09-22T13:16:31.783 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.763+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.94 2023-09-22T13:16:31.783 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.763+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.783 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.764+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.94 does not exist 2023-09-22T13:16:31.783 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.764+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.95 2023-09-22T13:16:31.783 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.764+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.783 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.764+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.95 does not exist 2023-09-22T13:16:31.783 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.764+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.96 2023-09-22T13:16:31.783 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.764+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.783 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.764+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.96 does not exist 2023-09-22T13:16:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.764+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.97 2023-09-22T13:16:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.764+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.764+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.97 does not exist 2023-09-22T13:16:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.764+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.98 2023-09-22T13:16:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.764+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.765+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.98 does not exist 2023-09-22T13:16:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.765+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.99 2023-09-22T13:16:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.765+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.765+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.99 does not exist 2023-09-22T13:16:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.765+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.100 2023-09-22T13:16:31.784 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.765+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.785 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.765+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.100 does not exist 2023-09-22T13:16:31.785 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.765+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.101 2023-09-22T13:16:31.785 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.765+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.785 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.765+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.101 does not exist 2023-09-22T13:16:31.785 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.765+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.102 2023-09-22T13:16:31.785 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.765+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.785 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.766+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.102 does not exist 2023-09-22T13:16:31.785 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.766+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.103 2023-09-22T13:16:31.785 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.766+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.785 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.766+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.103 does not exist 2023-09-22T13:16:31.785 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.766+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.104 2023-09-22T13:16:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.766+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.766+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.104 does not exist 2023-09-22T13:16:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.766+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.105 2023-09-22T13:16:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.766+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.766+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.105 does not exist 2023-09-22T13:16:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.766+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.106 2023-09-22T13:16:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.766+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.766+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.106 does not exist 2023-09-22T13:16:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.766+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.107 2023-09-22T13:16:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.766+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.767+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.107 does not exist 2023-09-22T13:16:31.786 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.767+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.108 2023-09-22T13:16:31.787 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.767+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.787 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.767+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.108 does not exist 2023-09-22T13:16:31.787 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.767+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.109 2023-09-22T13:16:31.787 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.767+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.787 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.767+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.109 does not exist 2023-09-22T13:16:31.787 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.767+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.110 2023-09-22T13:16:31.787 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.767+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.787 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.767+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.110 does not exist 2023-09-22T13:16:31.787 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.767+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.111 2023-09-22T13:16:31.787 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.767+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.787 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.767+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.111 does not exist 2023-09-22T13:16:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.767+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.112 2023-09-22T13:16:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.767+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.768+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.112 does not exist 2023-09-22T13:16:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.768+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.113 2023-09-22T13:16:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.768+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.768+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.113 does not exist 2023-09-22T13:16:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.768+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.114 2023-09-22T13:16:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.768+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.768+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.114 does not exist 2023-09-22T13:16:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.768+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.115 2023-09-22T13:16:31.788 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.768+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.768+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.115 does not exist 2023-09-22T13:16:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.768+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.116 2023-09-22T13:16:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.768+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.769+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.116 does not exist 2023-09-22T13:16:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.769+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.117 2023-09-22T13:16:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.769+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.769+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.117 does not exist 2023-09-22T13:16:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.769+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.118 2023-09-22T13:16:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.769+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.769+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.118 does not exist 2023-09-22T13:16:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.769+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.119 2023-09-22T13:16:31.789 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.769+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.790 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.769+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.119 does not exist 2023-09-22T13:16:31.790 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.769+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.120 2023-09-22T13:16:31.790 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.769+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.790 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.769+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.120 does not exist 2023-09-22T13:16:31.790 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.769+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.121 2023-09-22T13:16:31.790 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.769+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.790 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.770+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.121 does not exist 2023-09-22T13:16:31.790 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.770+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.122 2023-09-22T13:16:31.790 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.770+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.790 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.770+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.122 does not exist 2023-09-22T13:16:31.790 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.770+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.123 2023-09-22T13:16:31.791 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.770+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.791 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.770+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.123 does not exist 2023-09-22T13:16:31.791 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.770+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.124 2023-09-22T13:16:31.791 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.770+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.791 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.770+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.124 does not exist 2023-09-22T13:16:31.791 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.770+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.125 2023-09-22T13:16:31.791 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.770+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.791 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.771+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.125 does not exist 2023-09-22T13:16:31.791 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.771+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.126 2023-09-22T13:16:31.791 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.771+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.791 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.771+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.126 does not exist 2023-09-22T13:16:31.792 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.771+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):37 probing oid=data_log.127 2023-09-22T13:16:31.792 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.771+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.792 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.771+0000 7fb994a3f500 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):59: oid=data_log.127 does not exist 2023-09-22T13:16:31.792 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.771+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1206 entering 2023-09-22T13:16:31.792 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.772+0000 7fb994a3f500 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1164 entering 2023-09-22T13:16:31.792 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.774+0000 7fb9517e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:16:31.792 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.774+0000 7fb994a3f500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:16:31.792 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:31.774+0000 7fb994a3f500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:16:34.686 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.685+0000 7fb994a3f500 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-09-22T13:16:34.686 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.685+0000 7fb994a3f500 5 note: GC not initialized 2023-09-22T13:16:34.686 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.686+0000 7fb8b3fff700 20 reqs_thread_entry: start 2023-09-22T13:16:34.687 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.687+0000 7fb8b1ffb700 20 reqs_thread_entry: start 2023-09-22T13:16:34.687 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.687+0000 7fb994a3f500 20 init_complete bucket index max shards: 11 2023-09-22T13:16:34.687 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.687+0000 7fb994a3f500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:16:34.687 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.687+0000 7fb8a3fff700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:17:05 2023 2023-09-22T13:16:34.687 INFO:teuthology.orchestra.run.smithi096.stderr: 2023-09-22T13:16:34.688 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.687+0000 7fb8a37fe700 20 reqs_thread_entry: start 2023-09-22T13:16:34.709 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.709+0000 7fb994a3f500 20 remove_watcher() i=0 2023-09-22T13:16:34.709 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.709+0000 7fb994a3f500 2 removed watcher, disabling cache 2023-09-22T13:16:34.710 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.710+0000 7fb994a3f500 20 remove_watcher() i=1 2023-09-22T13:16:34.710 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.711+0000 7fb994a3f500 20 remove_watcher() i=2 2023-09-22T13:16:34.711 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.711+0000 7fb994a3f500 20 remove_watcher() i=3 2023-09-22T13:16:34.712 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.712+0000 7fb994a3f500 20 remove_watcher() i=4 2023-09-22T13:16:34.713 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.713+0000 7fb994a3f500 20 remove_watcher() i=5 2023-09-22T13:16:34.714 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.714+0000 7fb994a3f500 20 remove_watcher() i=6 2023-09-22T13:16:34.715 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.715+0000 7fb994a3f500 20 remove_watcher() i=7 2023-09-22T13:16:34.716 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.716+0000 7fb9517e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:16:34.719 INFO:teuthology.orchestra.run.smithi096.stdout:[] 2023-09-22T13:16:34.721 DEBUG:tasks.util.rgw: json result: [] 2023-09-22T13:16:34.721 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-22T13:16:34.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-22T13:16:34.722 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-09-22T13:16:34.770 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:16:34.771 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:16:34.777 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.777+0000 7fb0945f2500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:16:34.777 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.777+0000 7fb0945f2500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:16:34.781 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.781+0000 7fafd17fa700 20 reqs_thread_entry: start 2023-09-22T13:16:34.781 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.781+0000 7fb0945f2500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.782 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.782+0000 7fb0945f2500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:16:34.782 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.782+0000 7fb0945f2500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.783 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.782+0000 7fb0945f2500 20 rados_obj.operate() r=0 bl.length=873 2023-09-22T13:16:34.783 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.782+0000 7fb0945f2500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.783 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.783+0000 7fb0945f2500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:34.783 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.783+0000 7fb0945f2500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.784 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.784+0000 7fb0945f2500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:16:34.784 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.784+0000 7fb0945f2500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.784 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.784+0000 7fb0945f2500 20 rados_obj.operate() r=0 bl.length=358 2023-09-22T13:16:34.787 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2023-09-22T13:16:34.788 INFO:teuthology.orchestra.run.smithi096.stdout: "id": "42123edd-4a97-43ec-80bc-fc5350f88c11", 2023-09-22T13:16:34.788 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "default", 2023-09-22T13:16:34.788 INFO:teuthology.orchestra.run.smithi096.stdout: "domain_root": "default.rgw.meta:root", 2023-09-22T13:16:34.788 INFO:teuthology.orchestra.run.smithi096.stdout: "control_pool": "default.rgw.control", 2023-09-22T13:16:34.788 INFO:teuthology.orchestra.run.smithi096.stdout: "gc_pool": "default.rgw.log:gc", 2023-09-22T13:16:34.788 INFO:teuthology.orchestra.run.smithi096.stdout: "lc_pool": "default.rgw.log:lc", 2023-09-22T13:16:34.788 INFO:teuthology.orchestra.run.smithi096.stdout: "log_pool": "default.rgw.log", 2023-09-22T13:16:34.788 INFO:teuthology.orchestra.run.smithi096.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-09-22T13:16:34.788 INFO:teuthology.orchestra.run.smithi096.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-09-22T13:16:34.788 INFO:teuthology.orchestra.run.smithi096.stdout: "roles_pool": "default.rgw.meta:roles", 2023-09-22T13:16:34.789 INFO:teuthology.orchestra.run.smithi096.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-09-22T13:16:34.789 INFO:teuthology.orchestra.run.smithi096.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-09-22T13:16:34.789 INFO:teuthology.orchestra.run.smithi096.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-09-22T13:16:34.789 INFO:teuthology.orchestra.run.smithi096.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-09-22T13:16:34.789 INFO:teuthology.orchestra.run.smithi096.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-09-22T13:16:34.789 INFO:teuthology.orchestra.run.smithi096.stdout: "otp_pool": "default.rgw.otp", 2023-09-22T13:16:34.789 INFO:teuthology.orchestra.run.smithi096.stdout: "system_key": { 2023-09-22T13:16:34.789 INFO:teuthology.orchestra.run.smithi096.stdout: "access_key": "", 2023-09-22T13:16:34.789 INFO:teuthology.orchestra.run.smithi096.stdout: "secret_key": "" 2023-09-22T13:16:34.789 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2023-09-22T13:16:34.789 INFO:teuthology.orchestra.run.smithi096.stdout: "placement_pools": [ 2023-09-22T13:16:34.789 INFO:teuthology.orchestra.run.smithi096.stdout: { 2023-09-22T13:16:34.790 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "default-placement", 2023-09-22T13:16:34.790 INFO:teuthology.orchestra.run.smithi096.stdout: "val": { 2023-09-22T13:16:34.790 INFO:teuthology.orchestra.run.smithi096.stdout: "index_pool": "default.rgw.buckets.index", 2023-09-22T13:16:34.790 INFO:teuthology.orchestra.run.smithi096.stdout: "storage_classes": { 2023-09-22T13:16:34.790 INFO:teuthology.orchestra.run.smithi096.stdout: "STANDARD": { 2023-09-22T13:16:34.790 INFO:teuthology.orchestra.run.smithi096.stdout: "data_pool": "default.rgw.buckets.data", 2023-09-22T13:16:34.790 INFO:teuthology.orchestra.run.smithi096.stdout: "compression_type": "random" 2023-09-22T13:16:34.790 INFO:teuthology.orchestra.run.smithi096.stdout: } 2023-09-22T13:16:34.790 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2023-09-22T13:16:34.790 INFO:teuthology.orchestra.run.smithi096.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-09-22T13:16:34.790 INFO:teuthology.orchestra.run.smithi096.stdout: "index_type": 0 2023-09-22T13:16:34.791 INFO:teuthology.orchestra.run.smithi096.stdout: } 2023-09-22T13:16:34.791 INFO:teuthology.orchestra.run.smithi096.stdout: } 2023-09-22T13:16:34.791 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2023-09-22T13:16:34.791 INFO:teuthology.orchestra.run.smithi096.stdout: "realm_id": "", 2023-09-22T13:16:34.791 INFO:teuthology.orchestra.run.smithi096.stdout: "notif_pool": "default.rgw.log:notif" 2023-09-22T13:16:34.791 INFO:teuthology.orchestra.run.smithi096.stdout:} 2023-09-22T13:16:34.792 DEBUG:tasks.util.rgw: json result: {'id': '42123edd-4a97-43ec-80bc-fc5350f88c11', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-09-22T13:16:34.792 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2023-09-22T13:16:34.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2023-09-22T13:16:34.792 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2023-09-22T13:16:34.832 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:16:34.832 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:16:34.838 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.838+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.840 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.840+0000 7efd05046500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:34.840 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.840+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.840 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.840+0000 7efd05046500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:34.841 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.840+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.841 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.841+0000 7efd05046500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:34.841 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.841+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.842 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.842+0000 7efd05046500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:16:34.853 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.853+0000 7efd05046500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:16:34.853 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.853+0000 7efd05046500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:16:34.853 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.853+0000 7efd05046500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:16:34.854 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.853+0000 7efd05046500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:16:34.854 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.853+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.854 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.853+0000 7efd05046500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:16:34.854 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.853+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.856 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.853+0000 7efd05046500 20 rados_obj.operate() r=0 bl.length=889 2023-09-22T13:16:34.856 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.854+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.856 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.854+0000 7efd05046500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:16:34.856 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.854+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.856 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.854+0000 7efd05046500 20 rados_obj.operate() r=0 bl.length=358 2023-09-22T13:16:34.857 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.854+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.857 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.854+0000 7efd05046500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:34.857 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.854+0000 7efd05046500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:16:34.857 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.854+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.857 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.855+0000 7efd05046500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:34.857 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.855+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.857 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.855+0000 7efd05046500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:16:34.857 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.855+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.857 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.855+0000 7efd05046500 20 rados_obj.operate() r=0 bl.length=358 2023-09-22T13:16:34.857 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.855+0000 7efd05046500 20 zonegroup default 2023-09-22T13:16:34.857 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.855+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.857 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.855+0000 7efd05046500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:34.858 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.855+0000 7efd05046500 10 Cannot find current period zone using local zone 2023-09-22T13:16:34.858 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.855+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.858 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.856+0000 7efd05046500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:34.858 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.856+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.858 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.856+0000 7efd05046500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:16:34.858 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.856+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.859 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.856+0000 7efd05046500 20 rados_obj.operate() r=0 bl.length=889 2023-09-22T13:16:34.859 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.856+0000 7efd05046500 20 zone default found 2023-09-22T13:16:34.859 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.856+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.859 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.857+0000 7efd05046500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:34.859 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.857+0000 7efd05046500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.859 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.857+0000 7efd05046500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:34.860 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.857+0000 7efd05046500 20 started sync module instance, tier type = 2023-09-22T13:16:34.860 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.857+0000 7efd05046500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:16:34.864 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.864+0000 7efcf0528700 0 -- 172.21.15.96:0/1948430912 >> [v2:172.21.15.96:6824/140822,v1:172.21.15.96:6825/140822] conn(0x55b7caf1c6a0 msgr2=0x55b7caf1ea90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:16:34.864 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.864+0000 7efd05046500 20 add_watcher() i=0 2023-09-22T13:16:34.864 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.864+0000 7efd05046500 20 add_watcher() i=1 2023-09-22T13:16:34.864 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.864+0000 7efd05046500 20 add_watcher() i=2 2023-09-22T13:16:34.864 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.864+0000 7efd05046500 20 add_watcher() i=3 2023-09-22T13:16:34.865 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.864+0000 7efd05046500 20 add_watcher() i=4 2023-09-22T13:16:34.865 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.864+0000 7efd05046500 20 add_watcher() i=5 2023-09-22T13:16:34.865 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.864+0000 7efd05046500 20 add_watcher() i=6 2023-09-22T13:16:34.865 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.865+0000 7efd05046500 20 add_watcher() i=7 2023-09-22T13:16:34.865 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.865+0000 7efd05046500 2 all 8 watchers are set, enabling cache 2023-09-22T13:16:34.866 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.866+0000 7efcc1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:16:34.866 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.866+0000 7efd05046500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:16:34.866 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.866+0000 7efd05046500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:16:34.869 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.869+0000 7efcf0528700 0 -- 172.21.15.96:0/1235457447 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55b7caf39ef0 msgr2=0x55b7caf40530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:16:34.871 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.871+0000 7efd05046500 5 note: GC not initialized 2023-09-22T13:16:34.872 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.872+0000 7efc38ff9700 20 reqs_thread_entry: start 2023-09-22T13:16:34.872 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.872+0000 7efc2e7fc700 20 reqs_thread_entry: start 2023-09-22T13:16:34.872 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.872+0000 7efd05046500 20 init_complete bucket index max shards: 11 2023-09-22T13:16:34.873 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.872+0000 7efd05046500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:16:34.873 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.872+0000 7efc2cff9700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:17:05 2023 2023-09-22T13:16:34.873 INFO:teuthology.orchestra.run.smithi096.stderr: 2023-09-22T13:16:34.873 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.872+0000 7efc13fff700 20 reqs_thread_entry: start 2023-09-22T13:16:34.882 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.882+0000 7efd05046500 20 remove_watcher() i=0 2023-09-22T13:16:34.882 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.882+0000 7efd05046500 2 removed watcher, disabling cache 2023-09-22T13:16:34.882 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.882+0000 7efd05046500 20 remove_watcher() i=1 2023-09-22T13:16:34.883 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.883+0000 7efd05046500 20 remove_watcher() i=2 2023-09-22T13:16:34.884 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.884+0000 7efd05046500 20 remove_watcher() i=3 2023-09-22T13:16:34.885 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.885+0000 7efd05046500 20 remove_watcher() i=4 2023-09-22T13:16:34.886 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.886+0000 7efd05046500 20 remove_watcher() i=5 2023-09-22T13:16:34.886 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.886+0000 7efd05046500 20 remove_watcher() i=6 2023-09-22T13:16:34.887 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.887+0000 7efd05046500 20 remove_watcher() i=7 2023-09-22T13:16:34.888 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.888+0000 7efcc1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:16:34.891 INFO:teuthology.orchestra.run.smithi096.stdout:[] 2023-09-22T13:16:34.892 DEBUG:tasks.util.rgw: json result: [] 2023-09-22T13:16:34.892 INFO:tasks.rgw:Configuring storage class type = LUKEWARM 2023-09-22T13:16:34.892 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2023-09-22T13:16:34.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2023-09-22T13:16:34.893 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM 2023-09-22T13:16:34.978 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:16:34.978 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:16:34.984 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.984+0000 7fa094074500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:16:34.984 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.984+0000 7fa094074500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:16:34.988 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.988+0000 7f9fc3fff700 20 reqs_thread_entry: start 2023-09-22T13:16:34.988 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.988+0000 7fa094074500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.989 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.989+0000 7fa094074500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:34.989 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.989+0000 7fa094074500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.990 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.990+0000 7fa094074500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:16:34.990 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.990+0000 7fa094074500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.990 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.990+0000 7fa094074500 20 rados_obj.operate() r=0 bl.length=358 2023-09-22T13:16:34.991 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.990+0000 7fa094074500 20 rados->read ofs=0 len=0 2023-09-22T13:16:34.991 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:34.990+0000 7fa094074500 20 rados_obj.operate() r=0 bl.length=889 2023-09-22T13:16:34.994 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2023-09-22T13:16:34.994 INFO:teuthology.orchestra.run.smithi096.stdout: { 2023-09-22T13:16:34.994 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "default-placement", 2023-09-22T13:16:34.994 INFO:teuthology.orchestra.run.smithi096.stdout: "val": { 2023-09-22T13:16:34.995 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "default-placement", 2023-09-22T13:16:34.995 INFO:teuthology.orchestra.run.smithi096.stdout: "tags": [], 2023-09-22T13:16:34.995 INFO:teuthology.orchestra.run.smithi096.stdout: "storage_classes": [ 2023-09-22T13:16:34.995 INFO:teuthology.orchestra.run.smithi096.stdout: "LUKEWARM", 2023-09-22T13:16:34.995 INFO:teuthology.orchestra.run.smithi096.stdout: "STANDARD" 2023-09-22T13:16:34.995 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2023-09-22T13:16:34.995 INFO:teuthology.orchestra.run.smithi096.stdout: } 2023-09-22T13:16:34.996 INFO:teuthology.orchestra.run.smithi096.stdout: } 2023-09-22T13:16:34.996 INFO:teuthology.orchestra.run.smithi096.stdout:] 2023-09-22T13:16:34.997 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['LUKEWARM', 'STANDARD']}}] 2023-09-22T13:16:34.997 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2023-09-22T13:16:34.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2023-09-22T13:16:34.997 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM --data-pool default.rgw.buckets.data.lukewarm 2023-09-22T13:16:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:16:35.044 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:16:35.050 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.050+0000 7f968a5ff500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:16:35.050 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.050+0000 7f968a5ff500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:16:35.054 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.054+0000 7f95bf7fe700 20 reqs_thread_entry: start 2023-09-22T13:16:35.054 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.054+0000 7f968a5ff500 20 rados->read ofs=0 len=0 2023-09-22T13:16:35.055 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.055+0000 7f968a5ff500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:16:35.055 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.055+0000 7f968a5ff500 20 rados->read ofs=0 len=0 2023-09-22T13:16:35.055 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.055+0000 7f968a5ff500 20 rados_obj.operate() r=0 bl.length=889 2023-09-22T13:16:35.055 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.055+0000 7f968a5ff500 20 rados->read ofs=0 len=0 2023-09-22T13:16:35.056 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.056+0000 7f968a5ff500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:35.056 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.056+0000 7f968a5ff500 20 rados->read ofs=0 len=0 2023-09-22T13:16:35.056 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.056+0000 7f968a5ff500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:16:35.056 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.056+0000 7f968a5ff500 20 rados->read ofs=0 len=0 2023-09-22T13:16:35.057 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.056+0000 7f968a5ff500 20 rados_obj.operate() r=0 bl.length=382 2023-09-22T13:16:35.060 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2023-09-22T13:16:35.060 INFO:teuthology.orchestra.run.smithi096.stdout: "id": "42123edd-4a97-43ec-80bc-fc5350f88c11", 2023-09-22T13:16:35.060 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "default", 2023-09-22T13:16:35.060 INFO:teuthology.orchestra.run.smithi096.stdout: "domain_root": "default.rgw.meta:root", 2023-09-22T13:16:35.060 INFO:teuthology.orchestra.run.smithi096.stdout: "control_pool": "default.rgw.control", 2023-09-22T13:16:35.060 INFO:teuthology.orchestra.run.smithi096.stdout: "gc_pool": "default.rgw.log:gc", 2023-09-22T13:16:35.060 INFO:teuthology.orchestra.run.smithi096.stdout: "lc_pool": "default.rgw.log:lc", 2023-09-22T13:16:35.060 INFO:teuthology.orchestra.run.smithi096.stdout: "log_pool": "default.rgw.log", 2023-09-22T13:16:35.061 INFO:teuthology.orchestra.run.smithi096.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-09-22T13:16:35.061 INFO:teuthology.orchestra.run.smithi096.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-09-22T13:16:35.061 INFO:teuthology.orchestra.run.smithi096.stdout: "roles_pool": "default.rgw.meta:roles", 2023-09-22T13:16:35.061 INFO:teuthology.orchestra.run.smithi096.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-09-22T13:16:35.061 INFO:teuthology.orchestra.run.smithi096.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-09-22T13:16:35.061 INFO:teuthology.orchestra.run.smithi096.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-09-22T13:16:35.061 INFO:teuthology.orchestra.run.smithi096.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-09-22T13:16:35.061 INFO:teuthology.orchestra.run.smithi096.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-09-22T13:16:35.061 INFO:teuthology.orchestra.run.smithi096.stdout: "otp_pool": "default.rgw.otp", 2023-09-22T13:16:35.061 INFO:teuthology.orchestra.run.smithi096.stdout: "system_key": { 2023-09-22T13:16:35.061 INFO:teuthology.orchestra.run.smithi096.stdout: "access_key": "", 2023-09-22T13:16:35.061 INFO:teuthology.orchestra.run.smithi096.stdout: "secret_key": "" 2023-09-22T13:16:35.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2023-09-22T13:16:35.062 INFO:teuthology.orchestra.run.smithi096.stdout: "placement_pools": [ 2023-09-22T13:16:35.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2023-09-22T13:16:35.062 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "default-placement", 2023-09-22T13:16:35.062 INFO:teuthology.orchestra.run.smithi096.stdout: "val": { 2023-09-22T13:16:35.062 INFO:teuthology.orchestra.run.smithi096.stdout: "index_pool": "default.rgw.buckets.index", 2023-09-22T13:16:35.062 INFO:teuthology.orchestra.run.smithi096.stdout: "storage_classes": { 2023-09-22T13:16:35.062 INFO:teuthology.orchestra.run.smithi096.stdout: "LUKEWARM": { 2023-09-22T13:16:35.062 INFO:teuthology.orchestra.run.smithi096.stdout: "data_pool": "default.rgw.buckets.data.lukewarm" 2023-09-22T13:16:35.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2023-09-22T13:16:35.062 INFO:teuthology.orchestra.run.smithi096.stdout: "STANDARD": { 2023-09-22T13:16:35.062 INFO:teuthology.orchestra.run.smithi096.stdout: "data_pool": "default.rgw.buckets.data", 2023-09-22T13:16:35.062 INFO:teuthology.orchestra.run.smithi096.stdout: "compression_type": "random" 2023-09-22T13:16:35.063 INFO:teuthology.orchestra.run.smithi096.stdout: } 2023-09-22T13:16:35.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2023-09-22T13:16:35.063 INFO:teuthology.orchestra.run.smithi096.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-09-22T13:16:35.063 INFO:teuthology.orchestra.run.smithi096.stdout: "index_type": 0 2023-09-22T13:16:35.063 INFO:teuthology.orchestra.run.smithi096.stdout: } 2023-09-22T13:16:35.063 INFO:teuthology.orchestra.run.smithi096.stdout: } 2023-09-22T13:16:35.063 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2023-09-22T13:16:35.063 INFO:teuthology.orchestra.run.smithi096.stdout: "realm_id": "", 2023-09-22T13:16:35.063 INFO:teuthology.orchestra.run.smithi096.stdout: "notif_pool": "default.rgw.log:notif" 2023-09-22T13:16:35.063 INFO:teuthology.orchestra.run.smithi096.stdout:} 2023-09-22T13:16:35.064 DEBUG:tasks.util.rgw: json result: {'id': '42123edd-4a97-43ec-80bc-fc5350f88c11', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-09-22T13:16:35.064 INFO:tasks.rgw:Configuring storage class type = FROZEN 2023-09-22T13:16:35.064 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2023-09-22T13:16:35.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2023-09-22T13:16:35.065 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN 2023-09-22T13:16:35.110 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:16:35.110 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:16:35.116 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.116+0000 7feeffbb0500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:16:35.116 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.116+0000 7feeffbb0500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:16:35.120 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.120+0000 7fee33fff700 20 reqs_thread_entry: start 2023-09-22T13:16:35.120 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.120+0000 7feeffbb0500 20 rados->read ofs=0 len=0 2023-09-22T13:16:35.121 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.121+0000 7feeffbb0500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:35.121 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.121+0000 7feeffbb0500 20 rados->read ofs=0 len=0 2023-09-22T13:16:35.122 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.121+0000 7feeffbb0500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:16:35.122 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.121+0000 7feeffbb0500 20 rados->read ofs=0 len=0 2023-09-22T13:16:35.122 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.122+0000 7feeffbb0500 20 rados_obj.operate() r=0 bl.length=382 2023-09-22T13:16:35.122 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.122+0000 7feeffbb0500 20 rados->read ofs=0 len=0 2023-09-22T13:16:35.122 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.122+0000 7feeffbb0500 20 rados_obj.operate() r=0 bl.length=956 2023-09-22T13:16:35.126 INFO:teuthology.orchestra.run.smithi096.stdout:[ 2023-09-22T13:16:35.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2023-09-22T13:16:35.126 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "default-placement", 2023-09-22T13:16:35.126 INFO:teuthology.orchestra.run.smithi096.stdout: "val": { 2023-09-22T13:16:35.126 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "default-placement", 2023-09-22T13:16:35.126 INFO:teuthology.orchestra.run.smithi096.stdout: "tags": [], 2023-09-22T13:16:35.126 INFO:teuthology.orchestra.run.smithi096.stdout: "storage_classes": [ 2023-09-22T13:16:35.126 INFO:teuthology.orchestra.run.smithi096.stdout: "FROZEN", 2023-09-22T13:16:35.126 INFO:teuthology.orchestra.run.smithi096.stdout: "LUKEWARM", 2023-09-22T13:16:35.126 INFO:teuthology.orchestra.run.smithi096.stdout: "STANDARD" 2023-09-22T13:16:35.127 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2023-09-22T13:16:35.127 INFO:teuthology.orchestra.run.smithi096.stdout: } 2023-09-22T13:16:35.127 INFO:teuthology.orchestra.run.smithi096.stdout: } 2023-09-22T13:16:35.127 INFO:teuthology.orchestra.run.smithi096.stdout:] 2023-09-22T13:16:35.128 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'LUKEWARM', 'STANDARD']}}] 2023-09-22T13:16:35.128 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2023-09-22T13:16:35.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2023-09-22T13:16:35.128 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN --data-pool default.rgw.buckets.data.frozen 2023-09-22T13:16:35.175 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:16:35.175 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:16:35.181 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.181+0000 7f530bca0500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:16:35.181 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.181+0000 7f530bca0500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:16:35.185 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.185+0000 7f5280ff9700 20 reqs_thread_entry: start 2023-09-22T13:16:35.185 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.185+0000 7f530bca0500 20 rados->read ofs=0 len=0 2023-09-22T13:16:35.186 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.186+0000 7f530bca0500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:16:35.186 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.186+0000 7f530bca0500 20 rados->read ofs=0 len=0 2023-09-22T13:16:35.187 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.186+0000 7f530bca0500 20 rados_obj.operate() r=0 bl.length=956 2023-09-22T13:16:35.187 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.186+0000 7f530bca0500 20 rados->read ofs=0 len=0 2023-09-22T13:16:35.187 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.187+0000 7f530bca0500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:16:35.187 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.187+0000 7f530bca0500 20 rados->read ofs=0 len=0 2023-09-22T13:16:35.187 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.187+0000 7f530bca0500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:16:35.187 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.187+0000 7f530bca0500 20 rados->read ofs=0 len=0 2023-09-22T13:16:35.188 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:16:35.188+0000 7f530bca0500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:16:35.191 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2023-09-22T13:16:35.191 INFO:teuthology.orchestra.run.smithi096.stdout: "id": "42123edd-4a97-43ec-80bc-fc5350f88c11", 2023-09-22T13:16:35.191 INFO:teuthology.orchestra.run.smithi096.stdout: "name": "default", 2023-09-22T13:16:35.191 INFO:teuthology.orchestra.run.smithi096.stdout: "domain_root": "default.rgw.meta:root", 2023-09-22T13:16:35.191 INFO:teuthology.orchestra.run.smithi096.stdout: "control_pool": "default.rgw.control", 2023-09-22T13:16:35.192 INFO:teuthology.orchestra.run.smithi096.stdout: "gc_pool": "default.rgw.log:gc", 2023-09-22T13:16:35.192 INFO:teuthology.orchestra.run.smithi096.stdout: "lc_pool": "default.rgw.log:lc", 2023-09-22T13:16:35.192 INFO:teuthology.orchestra.run.smithi096.stdout: "log_pool": "default.rgw.log", 2023-09-22T13:16:35.192 INFO:teuthology.orchestra.run.smithi096.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-09-22T13:16:35.192 INFO:teuthology.orchestra.run.smithi096.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-09-22T13:16:35.192 INFO:teuthology.orchestra.run.smithi096.stdout: "roles_pool": "default.rgw.meta:roles", 2023-09-22T13:16:35.192 INFO:teuthology.orchestra.run.smithi096.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-09-22T13:16:35.192 INFO:teuthology.orchestra.run.smithi096.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-09-22T13:16:35.192 INFO:teuthology.orchestra.run.smithi096.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-09-22T13:16:35.192 INFO:teuthology.orchestra.run.smithi096.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-09-22T13:16:35.192 INFO:teuthology.orchestra.run.smithi096.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-09-22T13:16:35.192 INFO:teuthology.orchestra.run.smithi096.stdout: "otp_pool": "default.rgw.otp", 2023-09-22T13:16:35.193 INFO:teuthology.orchestra.run.smithi096.stdout: "system_key": { 2023-09-22T13:16:35.193 INFO:teuthology.orchestra.run.smithi096.stdout: "access_key": "", 2023-09-22T13:16:35.193 INFO:teuthology.orchestra.run.smithi096.stdout: "secret_key": "" 2023-09-22T13:16:35.193 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2023-09-22T13:16:35.193 INFO:teuthology.orchestra.run.smithi096.stdout: "placement_pools": [ 2023-09-22T13:16:35.193 INFO:teuthology.orchestra.run.smithi096.stdout: { 2023-09-22T13:16:35.193 INFO:teuthology.orchestra.run.smithi096.stdout: "key": "default-placement", 2023-09-22T13:16:35.193 INFO:teuthology.orchestra.run.smithi096.stdout: "val": { 2023-09-22T13:16:35.193 INFO:teuthology.orchestra.run.smithi096.stdout: "index_pool": "default.rgw.buckets.index", 2023-09-22T13:16:35.193 INFO:teuthology.orchestra.run.smithi096.stdout: "storage_classes": { 2023-09-22T13:16:35.193 INFO:teuthology.orchestra.run.smithi096.stdout: "FROZEN": { 2023-09-22T13:16:35.193 INFO:teuthology.orchestra.run.smithi096.stdout: "data_pool": "default.rgw.buckets.data.frozen" 2023-09-22T13:16:35.194 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2023-09-22T13:16:35.194 INFO:teuthology.orchestra.run.smithi096.stdout: "LUKEWARM": { 2023-09-22T13:16:35.194 INFO:teuthology.orchestra.run.smithi096.stdout: "data_pool": "default.rgw.buckets.data.lukewarm" 2023-09-22T13:16:35.194 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2023-09-22T13:16:35.194 INFO:teuthology.orchestra.run.smithi096.stdout: "STANDARD": { 2023-09-22T13:16:35.194 INFO:teuthology.orchestra.run.smithi096.stdout: "data_pool": "default.rgw.buckets.data", 2023-09-22T13:16:35.194 INFO:teuthology.orchestra.run.smithi096.stdout: "compression_type": "random" 2023-09-22T13:16:35.194 INFO:teuthology.orchestra.run.smithi096.stdout: } 2023-09-22T13:16:35.194 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2023-09-22T13:16:35.194 INFO:teuthology.orchestra.run.smithi096.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-09-22T13:16:35.194 INFO:teuthology.orchestra.run.smithi096.stdout: "index_type": 0 2023-09-22T13:16:35.194 INFO:teuthology.orchestra.run.smithi096.stdout: } 2023-09-22T13:16:35.194 INFO:teuthology.orchestra.run.smithi096.stdout: } 2023-09-22T13:16:35.195 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2023-09-22T13:16:35.195 INFO:teuthology.orchestra.run.smithi096.stdout: "realm_id": "", 2023-09-22T13:16:35.195 INFO:teuthology.orchestra.run.smithi096.stdout: "notif_pool": "default.rgw.log:notif" 2023-09-22T13:16:35.195 INFO:teuthology.orchestra.run.smithi096.stdout:} 2023-09-22T13:16:35.196 DEBUG:tasks.util.rgw: json result: {'id': '42123edd-4a97-43ec-80bc-fc5350f88c11', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-09-22T13:16:35.196 INFO:tasks.rgw:Starting rgw... 2023-09-22T13:16:35.196 INFO:tasks.rgw:rgw client.0 config is {} 2023-09-22T13:16:35.196 INFO:tasks.rgw:Using beast as radosgw frontend 2023-09-22T13:16:35.196 INFO:tasks.rgw.client.0:Restarting daemon 2023-09-22T13:16:35.196 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2023-09-22T13:16:35.198 INFO:tasks.rgw.client.0:Started 2023-09-22T13:16:35.198 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi096.front.sepia.ceph.com:80/ 2023-09-22T13:16:35.199 DEBUG:teuthology.orchestra.run.smithi096:> curl http://smithi096.front.sepia.ceph.com:80/ 2023-09-22T13:16:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-22T13:16:35.284 INFO:teuthology.orchestra.run.smithi096.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T13:16:35.284 INFO:teuthology.orchestra.run.smithi096.stderr: Dload Upload Total Spent Left Speed 2023-09-22T13:16:35.284 INFO:teuthology.orchestra.run.smithi096.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi096.front.sepia.ceph.com port 80: Connection refused 2023-09-22T13:16:36.285 DEBUG:teuthology.orchestra.run.smithi096:> curl http://smithi096.front.sepia.ceph.com:80/ 2023-09-22T13:16:36.310 INFO:teuthology.orchestra.run.smithi096.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-22T13:16:36.311 INFO:teuthology.orchestra.run.smithi096.stderr: Dload Upload Total Spent Left Speed 2023-09-22T13:16:36.312 INFO:teuthology.orchestra.run.smithi096.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2023-09-22T13:16:36.313 INFO:teuthology.orchestra.run.smithi096.stdout:anonymous 2023-09-22T13:16:36.313 INFO:teuthology.run_tasks:Running task workunit... 2023-09-22T13:16:36.328 INFO:tasks.workunit:Pulling workunits from ref 31c126b7c1c0c06b66d9d19c5bfc51c46b503392 2023-09-22T13:16:36.328 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-09-22T13:16:36.328 DEBUG:teuthology.orchestra.run.smithi096:> stat -- /home/ubuntu/cephtest/mnt.0 2023-09-22T13:16:36.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T13:16:36.375 INFO:teuthology.orchestra.run.smithi096.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-09-22T13:16:36.376 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-09-22T13:16:36.437 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-09-22T13:16:36.437 DEBUG:teuthology.orchestra.run.smithi096:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-09-22T13:16:36.497 INFO:tasks.workunit:timeout=3h 2023-09-22T13:16:36.497 INFO:tasks.workunit:cleanup=True 2023-09-22T13:16:36.498 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 31c126b7c1c0c06b66d9d19c5bfc51c46b503392 2023-09-22T13:16:36.557 INFO:tasks.workunit.client.0.smithi096.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-09-22T13:17:38.893 INFO:tasks.workunit.client.0.smithi096.stderr:Note: switching to '31c126b7c1c0c06b66d9d19c5bfc51c46b503392'. 2023-09-22T13:17:38.894 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:38.894 INFO:tasks.workunit.client.0.smithi096.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-09-22T13:17:38.894 INFO:tasks.workunit.client.0.smithi096.stderr:changes and commit them, and you can discard any commits you make in this 2023-09-22T13:17:38.894 INFO:tasks.workunit.client.0.smithi096.stderr:state without impacting any branches by switching back to a branch. 2023-09-22T13:17:38.895 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:38.895 INFO:tasks.workunit.client.0.smithi096.stderr:If you want to create a new branch to retain commits you create, you may 2023-09-22T13:17:38.895 INFO:tasks.workunit.client.0.smithi096.stderr:do so (now or later) by using -c with the switch command. Example: 2023-09-22T13:17:38.895 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:38.895 INFO:tasks.workunit.client.0.smithi096.stderr: git switch -c 2023-09-22T13:17:38.895 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:38.895 INFO:tasks.workunit.client.0.smithi096.stderr:Or undo this operation with: 2023-09-22T13:17:38.895 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:38.896 INFO:tasks.workunit.client.0.smithi096.stderr: git switch - 2023-09-22T13:17:38.896 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:38.896 INFO:tasks.workunit.client.0.smithi096.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-09-22T13:17:38.896 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:38.896 INFO:tasks.workunit.client.0.smithi096.stderr:HEAD is now at 31c126b7c1c rgw: radosgw-admin bucket check should only print index entries with --check-objects flag 2023-09-22T13:17:38.899 DEBUG:teuthology.orchestra.run.smithi096:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-09-22T13:17:39.121 INFO:tasks.workunit.client.0.smithi096.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-09-22T13:17:39.123 INFO:tasks.workunit.client.0.smithi096.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-09-22T13:17:39.124 INFO:tasks.workunit.client.0.smithi096.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-09-22T13:17:39.667 INFO:tasks.workunit.client.0.smithi096.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-09-22T13:17:39.733 INFO:tasks.workunit.client.0.smithi096.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-09-22T13:17:39.771 INFO:tasks.workunit.client.0.smithi096.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-09-22T13:17:39.773 INFO:tasks.workunit.client.0.smithi096.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-09-22T13:17:39.773 INFO:tasks.workunit.client.0.smithi096.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-09-22T13:17:39.809 INFO:tasks.workunit.client.0.smithi096.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-09-22T13:17:39.812 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:17:39.812 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-09-22T13:17:39.830 INFO:tasks.workunit:Running workunits matching rgw/run-bucket-check.sh on client.0... 2023-09-22T13:17:39.831 INFO:tasks.workunit:Running workunit rgw/run-bucket-check.sh... 2023-09-22T13:17:39.832 DEBUG:teuthology.orchestra.run.smithi096:workunit test rgw/run-bucket-check.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=31c126b7c1c0c06b66d9d19c5bfc51c46b503392 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-bucket-check.sh 2023-09-22T13:17:39.892 INFO:tasks.workunit.client.0.smithi096.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-bucket-check.sh 2023-09-22T13:17:39.893 INFO:tasks.workunit.client.0.smithi096.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2023-09-22T13:17:39.893 INFO:tasks.workunit.client.0.smithi096.stderr:+ python3 -m venv /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2023-09-22T13:17:41.380 INFO:tasks.workunit.client.0.smithi096.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin/activate 2023-09-22T13:17:41.380 INFO:tasks.workunit.client.0.smithi096.stderr:++ deactivate nondestructive 2023-09-22T13:17:41.380 INFO:tasks.workunit.client.0.smithi096.stderr:++ '[' -n '' ']' 2023-09-22T13:17:41.380 INFO:tasks.workunit.client.0.smithi096.stderr:++ '[' -n '' ']' 2023-09-22T13:17:41.380 INFO:tasks.workunit.client.0.smithi096.stderr:++ '[' -n /usr/bin/bash -o -n '' ']' 2023-09-22T13:17:41.380 INFO:tasks.workunit.client.0.smithi096.stderr:++ hash -r 2023-09-22T13:17:41.380 INFO:tasks.workunit.client.0.smithi096.stderr:++ '[' -n '' ']' 2023-09-22T13:17:41.380 INFO:tasks.workunit.client.0.smithi096.stderr:++ unset VIRTUAL_ENV 2023-09-22T13:17:41.381 INFO:tasks.workunit.client.0.smithi096.stderr:++ '[' '!' nondestructive = nondestructive ']' 2023-09-22T13:17:41.381 INFO:tasks.workunit.client.0.smithi096.stderr:++ VIRTUAL_ENV=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2023-09-22T13:17:41.381 INFO:tasks.workunit.client.0.smithi096.stderr:++ export VIRTUAL_ENV 2023-09-22T13:17:41.381 INFO:tasks.workunit.client.0.smithi096.stderr:++ _OLD_VIRTUAL_PATH=/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-09-22T13:17:41.382 INFO:tasks.workunit.client.0.smithi096.stderr:++ PATH=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-09-22T13:17:41.382 INFO:tasks.workunit.client.0.smithi096.stderr:++ export PATH 2023-09-22T13:17:41.382 INFO:tasks.workunit.client.0.smithi096.stderr:++ '[' -n '' ']' 2023-09-22T13:17:41.382 INFO:tasks.workunit.client.0.smithi096.stderr:++ '[' -z '' ']' 2023-09-22T13:17:41.382 INFO:tasks.workunit.client.0.smithi096.stderr:++ _OLD_VIRTUAL_PS1= 2023-09-22T13:17:41.382 INFO:tasks.workunit.client.0.smithi096.stderr:++ '[' 'x(rgw) ' '!=' x ']' 2023-09-22T13:17:41.382 INFO:tasks.workunit.client.0.smithi096.stderr:++ PS1='(rgw) ' 2023-09-22T13:17:41.382 INFO:tasks.workunit.client.0.smithi096.stderr:++ export PS1 2023-09-22T13:17:41.382 INFO:tasks.workunit.client.0.smithi096.stderr:++ '[' -n /usr/bin/bash -o -n '' ']' 2023-09-22T13:17:41.382 INFO:tasks.workunit.client.0.smithi096.stderr:++ hash -r 2023-09-22T13:17:41.382 INFO:tasks.workunit.client.0.smithi096.stderr:+ pip install pip --upgrade 2023-09-22T13:17:41.810 INFO:tasks.workunit.client.0.smithi096.stdout:Collecting pip 2023-09-22T13:17:41.933 INFO:tasks.workunit.client.0.smithi096.stdout: Downloading https://files.pythonhosted.org/packages/a4/6d/6463d49a933f547439d6b5b98b46af8742cc03ae83543e4d7688c2420f8b/pip-21.3.1-py3-none-any.whl (1.7MB) 2023-09-22T13:17:42.175 INFO:tasks.workunit.client.0.smithi096.stdout:Installing collected packages: pip 2023-09-22T13:17:42.175 INFO:tasks.workunit.client.0.smithi096.stdout: Found existing installation: pip 9.0.3 2023-09-22T13:17:42.312 INFO:tasks.workunit.client.0.smithi096.stdout: Uninstalling pip-9.0.3: 2023-09-22T13:17:42.348 INFO:tasks.workunit.client.0.smithi096.stdout: Successfully uninstalled pip-9.0.3 2023-09-22T13:17:42.986 INFO:tasks.workunit.client.0.smithi096.stdout:Successfully installed pip-21.3.1 2023-09-22T13:17:43.166 INFO:tasks.workunit.client.0.smithi096.stderr:You are using pip version 21.3.1, however version 23.2.1 is available. 2023-09-22T13:17:43.166 INFO:tasks.workunit.client.0.smithi096.stderr:You should consider upgrading via the 'pip install --upgrade pip' command. 2023-09-22T13:17:43.197 INFO:tasks.workunit.client.0.smithi096.stderr:+ pip install boto3 2023-09-22T13:17:44.470 INFO:tasks.workunit.client.0.smithi096.stdout:Collecting boto3 2023-09-22T13:17:44.526 INFO:tasks.workunit.client.0.smithi096.stdout: Downloading boto3-1.23.10-py3-none-any.whl (132 kB) 2023-09-22T13:17:44.657 INFO:tasks.workunit.client.0.smithi096.stdout:Collecting s3transfer<0.6.0,>=0.5.0 2023-09-22T13:17:44.674 INFO:tasks.workunit.client.0.smithi096.stdout: Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB) 2023-09-22T13:17:45.724 INFO:tasks.workunit.client.0.smithi096.stdout:Collecting botocore<1.27.0,>=1.26.10 2023-09-22T13:17:45.741 INFO:tasks.workunit.client.0.smithi096.stdout: Downloading botocore-1.26.10-py3-none-any.whl (8.8 MB) 2023-09-22T13:17:46.652 INFO:tasks.workunit.client.0.smithi096.stdout:Collecting jmespath<2.0.0,>=0.7.1 2023-09-22T13:17:46.669 INFO:tasks.workunit.client.0.smithi096.stdout: Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB) 2023-09-22T13:17:46.768 INFO:tasks.workunit.client.0.smithi096.stdout:Collecting python-dateutil<3.0.0,>=2.1 2023-09-22T13:17:46.786 INFO:tasks.workunit.client.0.smithi096.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-09-22T13:17:46.942 INFO:tasks.workunit.client.0.smithi096.stdout:Collecting urllib3<1.27,>=1.25.4 2023-09-22T13:17:46.960 INFO:tasks.workunit.client.0.smithi096.stdout: Downloading urllib3-1.26.16-py2.py3-none-any.whl (143 kB) 2023-09-22T13:17:47.258 INFO:tasks.workunit.client.0.smithi096.stdout:Collecting six>=1.5 2023-09-22T13:17:47.275 INFO:tasks.workunit.client.0.smithi096.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-09-22T13:17:47.352 INFO:tasks.workunit.client.0.smithi096.stdout:Installing collected packages: six, urllib3, python-dateutil, jmespath, botocore, s3transfer, boto3 2023-09-22T13:17:48.349 INFO:tasks.workunit.client.0.smithi096.stdout:Successfully installed boto3-1.23.10 botocore-1.26.10 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2 six-1.16.0 urllib3-1.26.16 2023-09-22T13:17:48.636 INFO:tasks.workunit.client.0.smithi096.stderr:+ /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin/python3 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_bucket_check.py 2023-09-22T13:17:48.769 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin user create --uid check-tester --display-name "Check Testing" --access-key OJODXSLNX4LUNHQG99PA --secret 3l6ffld34qaymfomuh832j94738aie2x4p2o8h6n 2023-09-22T13:17:48.798 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:48.798 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:48.805 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.805+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.807 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.807+0000 7ff8c3c19500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:48.807 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.807+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.808 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.808+0000 7ff8c3c19500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:48.808 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.808+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.808 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.808+0000 7ff8c3c19500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:48.808 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.808+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.809 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.809+0000 7ff8c3c19500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:48.820 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.820+0000 7ff8c3c19500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:48.820 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.820+0000 7ff8c3c19500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:48.820 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.820+0000 7ff8c3c19500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:48.820 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.820+0000 7ff8c3c19500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:48.821 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.820+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.821 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.820+0000 7ff8c3c19500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:48.821 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.820+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.821 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.820+0000 7ff8c3c19500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:48.821 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.820+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.823 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.820+0000 7ff8c3c19500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:48.823 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.820+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.823 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.821+0000 7ff8c3c19500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:48.823 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.821+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.823 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.821+0000 7ff8c3c19500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:48.823 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.821+0000 7ff8c3c19500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:48.823 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.821+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.823 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.821+0000 7ff8c3c19500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:48.824 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.821+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.824 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.822+0000 7ff8c3c19500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:48.824 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.822+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.824 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.822+0000 7ff8c3c19500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:48.824 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.822+0000 7ff8c3c19500 20 zonegroup default 2023-09-22T13:17:48.824 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.822+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.824 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.822+0000 7ff8c3c19500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:48.824 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.822+0000 7ff8c3c19500 10 Cannot find current period zone using local zone 2023-09-22T13:17:48.824 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.822+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.824 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.822+0000 7ff8c3c19500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:48.824 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.822+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.824 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.823+0000 7ff8c3c19500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:48.824 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.823+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.825 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.823+0000 7ff8c3c19500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:48.825 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.823+0000 7ff8c3c19500 20 zone default found 2023-09-22T13:17:48.825 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.823+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.825 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.823+0000 7ff8c3c19500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:48.825 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.823+0000 7ff8c3c19500 20 rados->read ofs=0 len=0 2023-09-22T13:17:48.826 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.823+0000 7ff8c3c19500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:48.826 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.823+0000 7ff8c3c19500 20 started sync module instance, tier type = 2023-09-22T13:17:48.826 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.823+0000 7ff8c3c19500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:48.829 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.829+0000 7ff8af0fb700 0 -- 172.21.15.96:0/541217410 >> [v2:172.21.15.96:6824/140822,v1:172.21.15.96:6825/140822] conn(0x55c841af8800 msgr2=0x55c841afac20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:48.830 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.830+0000 7ff8c3c19500 20 add_watcher() i=0 2023-09-22T13:17:48.830 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.830+0000 7ff8c3c19500 20 add_watcher() i=1 2023-09-22T13:17:48.830 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.830+0000 7ff8c3c19500 20 add_watcher() i=2 2023-09-22T13:17:48.830 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.830+0000 7ff8c3c19500 20 add_watcher() i=3 2023-09-22T13:17:48.831 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.830+0000 7ff8c3c19500 20 add_watcher() i=4 2023-09-22T13:17:48.831 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.830+0000 7ff8c3c19500 20 add_watcher() i=5 2023-09-22T13:17:48.831 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.830+0000 7ff8c3c19500 20 add_watcher() i=6 2023-09-22T13:17:48.831 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.831+0000 7ff8c3c19500 20 add_watcher() i=7 2023-09-22T13:17:48.831 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.831+0000 7ff8c3c19500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:48.832 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.832+0000 7ff880fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:48.833 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.832+0000 7ff8c3c19500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:48.833 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.832+0000 7ff8c3c19500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:48.838 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.838+0000 7ff8ae8fa700 0 -- 172.21.15.96:0/1406103286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c841b37460 msgr2=0x55c841b3ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-09-22T13:17:48.840 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.840+0000 7ff8c3c19500 5 note: GC not initialized 2023-09-22T13:17:48.841 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.840+0000 7ff7ef7fe700 20 reqs_thread_entry: start 2023-09-22T13:17:48.841 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.841+0000 7ff7ed7fa700 20 reqs_thread_entry: start 2023-09-22T13:17:48.841 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.841+0000 7ff8c3c19500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:48.842 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.841+0000 7ff8c3c19500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:48.842 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.841+0000 7ff7db7fe700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:19 2023 2023-09-22T13:17:48.842 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:48.842 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.841+0000 7ff7daffd700 20 reqs_thread_entry: start 2023-09-22T13:17:48.842 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.841+0000 7ff8c3c19500 20 get_system_obj_state: rctx=0x7ffc358bf518 obj=default.rgw.meta:users.uid:check-tester state=0x55c841b0f880 s->prefetch_data=0 2023-09-22T13:17:48.842 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.841+0000 7ff8c3c19500 10 cache get: name=default.rgw.meta+users.uid+check-tester : miss 2023-09-22T13:17:48.842 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.842+0000 7ff8c3c19500 10 cache put: name=default.rgw.meta+users.uid+check-tester info.flags=0x0 2023-09-22T13:17:48.843 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.842+0000 7ff8c3c19500 10 adding default.rgw.meta+users.uid+check-tester to cache LRU end 2023-09-22T13:17:48.843 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.842+0000 7ff8c3c19500 20 get_system_obj_state: rctx=0x7ffc358bf528 obj=default.rgw.meta:users.keys:OJODXSLNX4LUNHQG99PA state=0x55c841b0f880 s->prefetch_data=0 2023-09-22T13:17:48.843 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.842+0000 7ff8c3c19500 10 cache get: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA : miss 2023-09-22T13:17:48.844 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.842+0000 7ff8c3c19500 10 cache put: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA info.flags=0x0 2023-09-22T13:17:48.844 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.842+0000 7ff8c3c19500 10 adding default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA to cache LRU end 2023-09-22T13:17:48.844 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.842+0000 7ff8c3c19500 20 get_system_obj_state: rctx=0x7ffc358bee28 obj=default.rgw.meta:users.keys:OJODXSLNX4LUNHQG99PA state=0x55c841b0f880 s->prefetch_data=0 2023-09-22T13:17:48.844 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.842+0000 7ff8c3c19500 10 cache get: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA : hit (negative entry) 2023-09-22T13:17:48.844 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.842+0000 7ff8c3c19500 20 get_system_obj_state: rctx=0x7ffc358bf168 obj=default.rgw.meta:users.keys:OJODXSLNX4LUNHQG99PA state=0x55c841b0f880 s->prefetch_data=0 2023-09-22T13:17:48.844 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.842+0000 7ff8c3c19500 10 cache get: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA : hit (negative entry) 2023-09-22T13:17:48.844 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.843+0000 7ff8c3c19500 10 cache put: name=default.rgw.meta+users.uid+check-tester info.flags=0x17 2023-09-22T13:17:48.844 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.843+0000 7ff8c3c19500 10 moving default.rgw.meta+users.uid+check-tester to cache LRU end 2023-09-22T13:17:48.844 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.843+0000 7ff8c3c19500 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.uid:check-tester, ofs0, ns] 2023-09-22T13:17:48.844 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.844+0000 7ff8ad0f7700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888064 cookie 94318583897536 notifier 4542 bl.length()=559 2023-09-22T13:17:48.844 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.844+0000 7ff8ad0f7700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+check-tester info.flags=0x17 2023-09-22T13:17:48.845 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.844+0000 7ff8ad0f7700 10 rgw watcher librados: moving default.rgw.meta+users.uid+check-tester to cache LRU end 2023-09-22T13:17:48.845 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.845+0000 7ff8c3c19500 10 cache put: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA info.flags=0x7 2023-09-22T13:17:48.845 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.845+0000 7ff8c3c19500 10 moving default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA to cache LRU end 2023-09-22T13:17:48.845 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.845+0000 7ff8c3c19500 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.keys:OJODXSLNX4LUNHQG99PA, ofs0, ns] 2023-09-22T13:17:48.846 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.845+0000 7ff8ad0f7700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 146028888064 cookie 94318583889168 notifier 4542 bl.length()=186 2023-09-22T13:17:48.846 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.845+0000 7ff8ad0f7700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA info.flags=0x7 2023-09-22T13:17:48.846 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.845+0000 7ff8ad0f7700 10 rgw watcher librados: moving default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA to cache LRU end 2023-09-22T13:17:48.847 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.847+0000 7ff8c3c19500 20 remove_watcher() i=0 2023-09-22T13:17:48.847 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.847+0000 7ff8c3c19500 2 removed watcher, disabling cache 2023-09-22T13:17:48.848 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.848+0000 7ff8c3c19500 20 remove_watcher() i=1 2023-09-22T13:17:48.849 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.849+0000 7ff8c3c19500 20 remove_watcher() i=2 2023-09-22T13:17:48.850 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.850+0000 7ff8c3c19500 20 remove_watcher() i=3 2023-09-22T13:17:48.850 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.850+0000 7ff8c3c19500 20 remove_watcher() i=4 2023-09-22T13:17:48.851 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.851+0000 7ff8c3c19500 20 remove_watcher() i=5 2023-09-22T13:17:48.852 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.852+0000 7ff8c3c19500 20 remove_watcher() i=6 2023-09-22T13:17:48.853 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.853+0000 7ff8c3c19500 20 remove_watcher() i=7 2023-09-22T13:17:48.853 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:48.853+0000 7ff880fe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:49.089 INFO:tasks.workunit.client.0.smithi096.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib64/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/ 2023-09-22T13:17:49.089 INFO:tasks.workunit.client.0.smithi096.stderr: warnings.warn(warning, PythonDeprecationWarning) 2023-09-22T13:17:49.089 INFO:tasks.workunit.client.0.smithi096.stderr:TEST: bucket check unlinked does not report normal entries 2023-09-22T13:17:49.090 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:49.090 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --min-age-hours 0 --dump-keys 2023-09-22T13:17:49.116 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:49.116 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:49.124 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.124+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.125 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.125+0000 7fac9e2c3500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:49.126 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.125+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.126 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.126+0000 7fac9e2c3500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:49.126 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.126+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.127 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.126+0000 7fac9e2c3500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:49.127 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.126+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.127 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.127+0000 7fac9e2c3500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:49.138 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.138+0000 7fac9e2c3500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:49.138 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.138+0000 7fac9e2c3500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:49.139 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.138+0000 7fac9e2c3500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:49.139 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.138+0000 7fac9e2c3500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:49.139 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.138+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.140 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.139+0000 7fac9e2c3500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:49.140 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.139+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.140 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.139+0000 7fac9e2c3500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:49.140 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.139+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.140 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.139+0000 7fac9e2c3500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:49.140 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.139+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.140 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.139+0000 7fac9e2c3500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:49.140 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.139+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.140 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.140+0000 7fac9e2c3500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:49.141 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.140+0000 7fac9e2c3500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:49.141 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.140+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.141 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.140+0000 7fac9e2c3500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:49.141 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.140+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.142 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.140+0000 7fac9e2c3500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:49.142 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.140+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.143 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.141+0000 7fac9e2c3500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:49.143 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.141+0000 7fac9e2c3500 20 zonegroup default 2023-09-22T13:17:49.143 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.141+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.143 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.141+0000 7fac9e2c3500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:49.143 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.141+0000 7fac9e2c3500 10 Cannot find current period zone using local zone 2023-09-22T13:17:49.143 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.141+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.143 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.141+0000 7fac9e2c3500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:49.143 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.141+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.143 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.141+0000 7fac9e2c3500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:49.143 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.141+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.143 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.142+0000 7fac9e2c3500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:49.143 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.142+0000 7fac9e2c3500 20 zone default found 2023-09-22T13:17:49.144 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.142+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.144 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.142+0000 7fac9e2c3500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:49.144 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.142+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.144 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.142+0000 7fac9e2c3500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:49.144 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.142+0000 7fac9e2c3500 20 started sync module instance, tier type = 2023-09-22T13:17:49.144 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.142+0000 7fac9e2c3500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:49.149 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.149+0000 7fac897a5700 0 -- 172.21.15.96:0/1559853968 >> [v2:172.21.15.96:6807/140797,v1:172.21.15.96:6809/140797] conn(0x7fac7c033910 msgr2=0x7fac7c032390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:49.149 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.149+0000 7fac9e2c3500 20 add_watcher() i=0 2023-09-22T13:17:49.149 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.149+0000 7fac9e2c3500 20 add_watcher() i=1 2023-09-22T13:17:49.149 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.149+0000 7fac9e2c3500 20 add_watcher() i=2 2023-09-22T13:17:49.149 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.149+0000 7fac9e2c3500 20 add_watcher() i=3 2023-09-22T13:17:49.149 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.149+0000 7fac9e2c3500 20 add_watcher() i=4 2023-09-22T13:17:49.150 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.149+0000 7fac9e2c3500 20 add_watcher() i=5 2023-09-22T13:17:49.150 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.149+0000 7fac9e2c3500 20 add_watcher() i=6 2023-09-22T13:17:49.150 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.150+0000 7fac9e2c3500 20 add_watcher() i=7 2023-09-22T13:17:49.150 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.150+0000 7fac9e2c3500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:49.151 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.151+0000 7fac5a7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:49.152 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.152+0000 7fac9e2c3500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:49.152 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.152+0000 7fac9e2c3500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:49.156 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.155+0000 7fac89fa6700 0 -- 172.21.15.96:0/2076714445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558ac31a7cd0 msgr2=0x558ac31aede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:49.356 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.356+0000 7fac89fa6700 0 --2- 172.21.15.96:0/2076714445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x558ac31a7cd0 0x558ac31aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-09-22T13:17:49.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.358+0000 7fac9e2c3500 5 note: GC not initialized 2023-09-22T13:17:49.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.359+0000 7fabc97fa700 20 reqs_thread_entry: start 2023-09-22T13:17:49.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.359+0000 7fabc2ffd700 20 reqs_thread_entry: start 2023-09-22T13:17:49.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.359+0000 7fac9e2c3500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:49.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.359+0000 7fac9e2c3500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:49.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.360+0000 7fabc17fa700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:19 2023 2023-09-22T13:17:49.360 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:49.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.360+0000 7fabc0ff9700 20 reqs_thread_entry: start 2023-09-22T13:17:49.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.360+0000 7fac9e2c3500 20 get_system_obj_state: rctx=0x7ffe06068378 obj=default.rgw.meta:root:check-bucket state=0x558ac3116010 s->prefetch_data=0 2023-09-22T13:17:49.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.360+0000 7fac9e2c3500 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-09-22T13:17:49.362 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.360+0000 7fac9e2c3500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x16 2023-09-22T13:17:49.362 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.360+0000 7fac9e2c3500 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:49.362 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.360+0000 7fac9e2c3500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:49.362 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.360+0000 7fac9e2c3500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:49.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.360+0000 7fac9e2c3500 10 cache get: name=default.rgw.meta+root+check-bucket : type miss (requested=0x11, cached=0x16) 2023-09-22T13:17:49.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.360+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.360+0000 7fac9e2c3500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:49.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.360+0000 7fac9e2c3500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-09-22T13:17:49.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.360+0000 7fac9e2c3500 10 moving default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:49.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.360+0000 7fac9e2c3500 20 get_system_obj_state: rctx=0x7ffe06068448 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x558ac3116010 s->prefetch_data=0 2023-09-22T13:17:49.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.360+0000 7fac9e2c3500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : miss 2023-09-22T13:17:49.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.361+0000 7fac9e2c3500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x16 2023-09-22T13:17:49.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.361+0000 7fac9e2c3500 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:49.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.361+0000 7fac9e2c3500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:49.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.361+0000 7fac9e2c3500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:49.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.361+0000 7fac9e2c3500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:49.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.361+0000 7fac9e2c3500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:49.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.361+0000 7fac9e2c3500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:49.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.361+0000 7fac9e2c3500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : type miss (requested=0x13, cached=0x16) 2023-09-22T13:17:49.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.361+0000 7fac9e2c3500 20 rados->read ofs=0 len=0 2023-09-22T13:17:49.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.361+0000 7fac9e2c3500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:49.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.361+0000 7fac9e2c3500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x13 2023-09-22T13:17:49.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.361+0000 7fac9e2c3500 10 moving default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:49.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.361+0000 7fac9e2c3500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:49.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.361+0000 7fac9e2c3500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:49.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.361+0000 7fac9e2c3500 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 2023-09-22T13:17:49.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.361+0000 7fac9e2c3500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:49.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.362+0000 7fac9e2c3500 1 NOTICE: finished shard 0 (0 entries found) 2023-09-22T13:17:49.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.362+0000 7fac9e2c3500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:49.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.363+0000 7fac9e2c3500 1 NOTICE: finished shard 1 (0 entries found) 2023-09-22T13:17:49.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.363+0000 7fac9e2c3500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:49.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.363+0000 7fac9e2c3500 1 NOTICE: finished shard 2 (0 entries found) 2023-09-22T13:17:49.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.363+0000 7fac9e2c3500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:49.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.364+0000 7fac9e2c3500 1 NOTICE: finished shard 3 (0 entries found) 2023-09-22T13:17:49.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.364+0000 7fac9e2c3500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:49.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.364+0000 7fac9e2c3500 1 NOTICE: finished shard 4 (0 entries found) 2023-09-22T13:17:49.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.364+0000 7fac9e2c3500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:49.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.365+0000 7fac9e2c3500 1 NOTICE: finished shard 5 (0 entries found) 2023-09-22T13:17:49.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.365+0000 7fac9e2c3500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:49.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.365+0000 7fac9e2c3500 1 NOTICE: finished shard 6 (0 entries found) 2023-09-22T13:17:49.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.365+0000 7fac9e2c3500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:49.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.366+0000 7fac9e2c3500 1 NOTICE: finished shard 7 (0 entries found) 2023-09-22T13:17:49.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.366+0000 7fac9e2c3500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:49.368 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.366+0000 7fac9e2c3500 1 NOTICE: finished shard 8 (0 entries found) 2023-09-22T13:17:49.368 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.366+0000 7fac9e2c3500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:49.368 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.367+0000 7fac9e2c3500 1 NOTICE: finished shard 9 (0 entries found) 2023-09-22T13:17:49.368 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.367+0000 7fac9e2c3500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:49.368 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.368+0000 7fac9e2c3500 1 NOTICE: finished shard 10 (0 entries found) 2023-09-22T13:17:49.368 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.368+0000 7fac9e2c3500 1 NOTICE: finished all shards (0 entries found) 2023-09-22T13:17:49.369 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.369+0000 7fac9e2c3500 20 remove_watcher() i=0 2023-09-22T13:17:49.369 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.369+0000 7fac9e2c3500 2 removed watcher, disabling cache 2023-09-22T13:17:49.370 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.370+0000 7fac9e2c3500 20 remove_watcher() i=1 2023-09-22T13:17:49.370 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.370+0000 7fac9e2c3500 20 remove_watcher() i=2 2023-09-22T13:17:49.371 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.371+0000 7fac9e2c3500 20 remove_watcher() i=3 2023-09-22T13:17:49.373 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.372+0000 7fac9e2c3500 20 remove_watcher() i=4 2023-09-22T13:17:49.374 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.373+0000 7fac9e2c3500 20 remove_watcher() i=5 2023-09-22T13:17:49.374 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.373+0000 7fac9e2c3500 20 remove_watcher() i=6 2023-09-22T13:17:49.374 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.374+0000 7fac9e2c3500 20 remove_watcher() i=7 2023-09-22T13:17:49.375 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:49.375+0000 7fac5a7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:49.381 INFO:tasks.workunit.client.0.smithi096.stderr:ceph config set client rgw_debug_inject_set_olh_err 2 2023-09-22T13:17:49.772 INFO:tasks.workunit.client.0.smithi096.stderr:ceph config set client rgw_debug_inject_olh_cancel_modification_err true 2023-09-22T13:17:51.221 INFO:tasks.workunit.client.0.smithi096.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-09-22T13:17:57.432 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bi list --bucket check-bucket --object c 2023-09-22T13:17:57.605 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:57.605 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:57.605 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.257+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.605 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.259+0000 7f150134c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.605 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.259+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.605 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.260+0000 7f150134c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.605 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.260+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.605 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.260+0000 7f150134c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.606 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.260+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.606 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.261+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.606 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.273+0000 7f150134c500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:57.606 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.273+0000 7f150134c500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:57.606 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.273+0000 7f150134c500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:57.606 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.273+0000 7f150134c500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:57.606 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.273+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.606 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.274+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.606 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.274+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.606 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.274+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:57.606 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.274+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.607 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.274+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.607 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.274+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.607 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.274+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:57.607 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.274+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.607 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.275+0000 7f150134c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.607 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.275+0000 7f150134c500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:57.607 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.275+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.607 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.275+0000 7f150134c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.607 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.275+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.607 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.275+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.607 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.275+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.607 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.275+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:57.607 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.275+0000 7f150134c500 20 zonegroup default 2023-09-22T13:17:57.608 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.275+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.608 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.276+0000 7f150134c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.608 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.276+0000 7f150134c500 10 Cannot find current period zone using local zone 2023-09-22T13:17:57.608 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.276+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.608 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.276+0000 7f150134c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.608 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.276+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.608 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.276+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.608 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.276+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.608 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.276+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:57.608 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.276+0000 7f150134c500 20 zone default found 2023-09-22T13:17:57.608 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.276+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.608 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.277+0000 7f150134c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.608 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.277+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.609 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.277+0000 7f150134c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.609 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.277+0000 7f150134c500 20 started sync module instance, tier type = 2023-09-22T13:17:57.609 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.277+0000 7f150134c500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:57.609 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.284+0000 7f150134c500 20 add_watcher() i=0 2023-09-22T13:17:57.609 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.284+0000 7f150134c500 20 add_watcher() i=1 2023-09-22T13:17:57.609 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.284+0000 7f150134c500 20 add_watcher() i=2 2023-09-22T13:17:57.609 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.284+0000 7f150134c500 20 add_watcher() i=3 2023-09-22T13:17:57.609 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.284+0000 7f150134c500 20 add_watcher() i=4 2023-09-22T13:17:57.609 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.284+0000 7f150134c500 20 add_watcher() i=5 2023-09-22T13:17:57.609 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.284+0000 7f150134c500 20 add_watcher() i=6 2023-09-22T13:17:57.609 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.285+0000 7f150134c500 20 add_watcher() i=7 2023-09-22T13:17:57.609 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.285+0000 7f150134c500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:57.609 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.286+0000 7f14e7fff700 0 -- 172.21.15.96:0/1282452050 >> [v2:172.21.15.96:6816/140810,v1:172.21.15.96:6817/140810] conn(0x55955272e800 msgr2=0x559552730bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:57.610 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.287+0000 7f14bdfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:57.610 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.288+0000 7f150134c500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:57.610 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.288+0000 7f150134c500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:57.610 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.292+0000 7f14ed02f700 0 -- 172.21.15.96:0/679333232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x559552768a20 msgr2=0x559552785af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:57.610 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.295+0000 7f150134c500 5 note: GC not initialized 2023-09-22T13:17:57.610 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.296+0000 7f1434ff9700 20 reqs_thread_entry: start 2023-09-22T13:17:57.610 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.296+0000 7f14227fc700 20 reqs_thread_entry: start 2023-09-22T13:17:57.610 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.296+0000 7f150134c500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:57.610 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.296+0000 7f150134c500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:57.610 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.296+0000 7f1420ff9700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:21 2023 2023-09-22T13:17:57.610 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:57.610 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.296+0000 7f140bfff700 20 reqs_thread_entry: start 2023-09-22T13:17:57.610 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.296+0000 7f150134c500 20 get_system_obj_state: rctx=0x7ffc6e021100 obj=default.rgw.meta:root:check-bucket state=0x5595526e5240 s->prefetch_data=0 2023-09-22T13:17:57.611 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.297+0000 7f150134c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.611 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.297+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.611 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.297+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:57.611 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.297+0000 7f150134c500 20 get_system_obj_state: rctx=0x7ffc6e021100 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x559552662820 s->prefetch_data=0 2023-09-22T13:17:57.611 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.298+0000 7f150134c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.611 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.298+0000 7f150134c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.611 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.298+0000 7f150134c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.611 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.298+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.611 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.298+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.611 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.298+0000 7f150134c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.611 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.298+0000 7f150134c500 20 get_system_obj_state: rctx=0x7ffc6e020ed0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x559552662820 s->prefetch_data=0 2023-09-22T13:17:57.611 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.298+0000 7f150134c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.611 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.298+0000 7f150134c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.612 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.298+0000 7f150134c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.612 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.298+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.612 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.299+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.612 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.299+0000 7f150134c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.612 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.299+0000 7f150134c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:57.612 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.299+0000 7f150134c500 20 get_system_obj_state: rctx=0x7ffc6e020ed0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x559552662820 s->prefetch_data=0 2023-09-22T13:17:57.612 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.300+0000 7f150134c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.612 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.300+0000 7f150134c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.612 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.300+0000 7f150134c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.612 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.300+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.612 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.300+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.612 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.300+0000 7f150134c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.612 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.300+0000 7f150134c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:57.613 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.301+0000 7f150134c500 20 get_system_obj_state: rctx=0x7ffc6e020ed0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x559552662820 s->prefetch_data=0 2023-09-22T13:17:57.613 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.301+0000 7f150134c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.613 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.301+0000 7f150134c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.613 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.301+0000 7f150134c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.613 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.301+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.613 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.301+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.613 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.301+0000 7f150134c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.613 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.301+0000 7f150134c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:57.613 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.302+0000 7f150134c500 20 get_system_obj_state: rctx=0x7ffc6e020ed0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x559552662820 s->prefetch_data=0 2023-09-22T13:17:57.613 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.302+0000 7f150134c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.613 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.302+0000 7f150134c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.613 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.302+0000 7f150134c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.613 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.302+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.614 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.303+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.614 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.303+0000 7f150134c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.614 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.303+0000 7f150134c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:57.614 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.303+0000 7f150134c500 20 get_system_obj_state: rctx=0x7ffc6e020ed0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x559552662820 s->prefetch_data=0 2023-09-22T13:17:57.614 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.304+0000 7f150134c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.614 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.304+0000 7f150134c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.614 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.304+0000 7f150134c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.614 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.304+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.614 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.304+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.614 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.304+0000 7f150134c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.614 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.304+0000 7f150134c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:57.615 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.304+0000 7f150134c500 20 get_system_obj_state: rctx=0x7ffc6e020ed0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x559552662820 s->prefetch_data=0 2023-09-22T13:17:57.615 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.305+0000 7f150134c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.615 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.305+0000 7f150134c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.615 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.305+0000 7f150134c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.615 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.305+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.615 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.305+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.615 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.305+0000 7f150134c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.615 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.305+0000 7f150134c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:57.615 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.306+0000 7f150134c500 20 get_system_obj_state: rctx=0x7ffc6e020ed0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x559552662820 s->prefetch_data=0 2023-09-22T13:17:57.615 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.306+0000 7f150134c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.615 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.306+0000 7f150134c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.615 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.306+0000 7f150134c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.615 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.306+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.616 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.307+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.616 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.307+0000 7f150134c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.616 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.307+0000 7f150134c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:57.616 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.307+0000 7f150134c500 20 get_system_obj_state: rctx=0x7ffc6e020ed0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x559552662820 s->prefetch_data=0 2023-09-22T13:17:57.616 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.308+0000 7f150134c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.644 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.308+0000 7f150134c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.644 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.308+0000 7f150134c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.644 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.308+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.644 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.308+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.644 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.308+0000 7f150134c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.644 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.308+0000 7f150134c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:57.644 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.308+0000 7f150134c500 20 get_system_obj_state: rctx=0x7ffc6e020ed0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x559552662820 s->prefetch_data=0 2023-09-22T13:17:57.644 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.309+0000 7f150134c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.645 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.309+0000 7f150134c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.645 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.309+0000 7f150134c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.645 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.309+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.645 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.309+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.645 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.309+0000 7f150134c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.645 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.309+0000 7f150134c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:57.645 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.310+0000 7f150134c500 20 get_system_obj_state: rctx=0x7ffc6e020ed0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x559552662820 s->prefetch_data=0 2023-09-22T13:17:57.645 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.310+0000 7f150134c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.645 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.310+0000 7f150134c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.645 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.310+0000 7f150134c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.645 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.310+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.645 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.311+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.645 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.311+0000 7f150134c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.646 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.311+0000 7f150134c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:57.646 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.311+0000 7f150134c500 20 get_system_obj_state: rctx=0x7ffc6e020ed0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x559552662820 s->prefetch_data=0 2023-09-22T13:17:57.646 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.311+0000 7f150134c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.646 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.311+0000 7f150134c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.646 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.311+0000 7f150134c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.646 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.311+0000 7f150134c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.646 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.312+0000 7f150134c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.646 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.312+0000 7f150134c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.646 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.312+0000 7f150134c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:57.646 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.314+0000 7f150134c500 20 remove_watcher() i=0 2023-09-22T13:17:57.646 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.314+0000 7f150134c500 2 removed watcher, disabling cache 2023-09-22T13:17:57.647 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.314+0000 7f150134c500 20 remove_watcher() i=1 2023-09-22T13:17:57.647 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.316+0000 7f150134c500 20 remove_watcher() i=2 2023-09-22T13:17:57.647 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.316+0000 7f150134c500 20 remove_watcher() i=3 2023-09-22T13:17:57.647 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.317+0000 7f150134c500 20 remove_watcher() i=4 2023-09-22T13:17:57.647 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.318+0000 7f150134c500 20 remove_watcher() i=5 2023-09-22T13:17:57.647 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.319+0000 7f150134c500 20 remove_watcher() i=6 2023-09-22T13:17:57.647 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.319+0000 7f150134c500 20 remove_watcher() i=7 2023-09-22T13:17:57.647 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.320+0000 7f14bdfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:57.647 INFO:tasks.workunit.client.0.smithi096.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-09-22T13:17:57.647 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bi list --bucket check-bucket --object d 2023-09-22T13:17:57.647 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:57.647 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:57.647 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.414+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.647 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.416+0000 7fc2d68ed500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.648 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.416+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.648 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.417+0000 7fc2d68ed500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.648 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.417+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.648 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.417+0000 7fc2d68ed500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.648 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.417+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.648 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.418+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.648 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.428+0000 7fc2d68ed500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:57.648 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.428+0000 7fc2d68ed500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:57.648 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.428+0000 7fc2d68ed500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:57.648 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.428+0000 7fc2d68ed500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:57.648 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.428+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.648 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.429+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.648 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.429+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.429+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:57.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.429+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.429+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.429+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.430+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:57.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.430+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.430+0000 7fc2d68ed500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.430+0000 7fc2d68ed500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:57.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.430+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.430+0000 7fc2d68ed500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.430+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.430+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.430+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.431+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:57.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.431+0000 7fc2d68ed500 20 zonegroup default 2023-09-22T13:17:57.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.431+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.431+0000 7fc2d68ed500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.431+0000 7fc2d68ed500 10 Cannot find current period zone using local zone 2023-09-22T13:17:57.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.431+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.431+0000 7fc2d68ed500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.431+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.431+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.431+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.432+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:57.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.432+0000 7fc2d68ed500 20 zone default found 2023-09-22T13:17:57.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.432+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.432+0000 7fc2d68ed500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.432+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.432+0000 7fc2d68ed500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.432+0000 7fc2d68ed500 20 started sync module instance, tier type = 2023-09-22T13:17:57.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.432+0000 7fc2d68ed500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:57.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.438+0000 7fc2c1dcf700 0 -- 172.21.15.96:0/1215819524 >> [v2:172.21.15.96:6824/140822,v1:172.21.15.96:6825/140822] conn(0x55eb94feb9b0 msgr2=0x55eb94fedde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:57.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.439+0000 7fc2d68ed500 20 add_watcher() i=0 2023-09-22T13:17:57.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.439+0000 7fc2d68ed500 20 add_watcher() i=1 2023-09-22T13:17:57.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.439+0000 7fc2d68ed500 20 add_watcher() i=2 2023-09-22T13:17:57.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.439+0000 7fc2d68ed500 20 add_watcher() i=3 2023-09-22T13:17:57.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.439+0000 7fc2d68ed500 20 add_watcher() i=4 2023-09-22T13:17:57.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.439+0000 7fc2d68ed500 20 add_watcher() i=5 2023-09-22T13:17:57.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.439+0000 7fc2d68ed500 20 add_watcher() i=6 2023-09-22T13:17:57.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.439+0000 7fc2d68ed500 20 add_watcher() i=7 2023-09-22T13:17:57.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.439+0000 7fc2d68ed500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:57.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.441+0000 7fc292fe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:57.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.441+0000 7fc2d68ed500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:57.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.441+0000 7fc2d68ed500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:57.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.444+0000 7fc2c1dcf700 0 -- 172.21.15.96:0/2436507364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55eb9502de50 msgr2=0x55eb95029f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-09-22T13:17:57.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.446+0000 7fc2d68ed500 5 note: GC not initialized 2023-09-22T13:17:57.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.447+0000 7fc201ffb700 20 reqs_thread_entry: start 2023-09-22T13:17:57.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.448+0000 7fc1f37fe700 20 reqs_thread_entry: start 2023-09-22T13:17:57.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.448+0000 7fc2d68ed500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:57.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.448+0000 7fc2d68ed500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:57.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.448+0000 7fc1f1ffb700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:21 2023 2023-09-22T13:17:57.652 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:57.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.448+0000 7fc1f17fa700 20 reqs_thread_entry: start 2023-09-22T13:17:57.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.448+0000 7fc2d68ed500 20 get_system_obj_state: rctx=0x7fff9aeb2c50 obj=default.rgw.meta:root:check-bucket state=0x55eb95002690 s->prefetch_data=0 2023-09-22T13:17:57.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.448+0000 7fc2d68ed500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.448+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.448+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:57.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.448+0000 7fc2d68ed500 20 get_system_obj_state: rctx=0x7fff9aeb2c50 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55eb94e7b810 s->prefetch_data=0 2023-09-22T13:17:57.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.449+0000 7fc2d68ed500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.449+0000 7fc2d68ed500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.449+0000 7fc2d68ed500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.449+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.449+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.449+0000 7fc2d68ed500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.449+0000 7fc2d68ed500 20 get_system_obj_state: rctx=0x7fff9aeb2a20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55eb94e7b810 s->prefetch_data=0 2023-09-22T13:17:57.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.450+0000 7fc2d68ed500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.450+0000 7fc2d68ed500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.450+0000 7fc2d68ed500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.450+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.450+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.450+0000 7fc2d68ed500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.450+0000 7fc2d68ed500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:57.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.450+0000 7fc2d68ed500 20 get_system_obj_state: rctx=0x7fff9aeb2a20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55eb94e7b810 s->prefetch_data=0 2023-09-22T13:17:57.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.451+0000 7fc2d68ed500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.451+0000 7fc2d68ed500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.451+0000 7fc2d68ed500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.451+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.451+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.451+0000 7fc2d68ed500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.451+0000 7fc2d68ed500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:57.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.452+0000 7fc2d68ed500 20 get_system_obj_state: rctx=0x7fff9aeb2a20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55eb94e7b810 s->prefetch_data=0 2023-09-22T13:17:57.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.452+0000 7fc2d68ed500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.452+0000 7fc2d68ed500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.452+0000 7fc2d68ed500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.452+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.453+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.453+0000 7fc2d68ed500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.453+0000 7fc2d68ed500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:57.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.453+0000 7fc2d68ed500 20 get_system_obj_state: rctx=0x7fff9aeb2a20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55eb94e7b810 s->prefetch_data=0 2023-09-22T13:17:57.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.453+0000 7fc2d68ed500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.453+0000 7fc2d68ed500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.453+0000 7fc2d68ed500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.453+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.454+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.454+0000 7fc2d68ed500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.454+0000 7fc2d68ed500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:57.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.454+0000 7fc2d68ed500 20 get_system_obj_state: rctx=0x7fff9aeb2a20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55eb94e7b810 s->prefetch_data=0 2023-09-22T13:17:57.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.455+0000 7fc2d68ed500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.455+0000 7fc2d68ed500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.455+0000 7fc2d68ed500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.455+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.455+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.455+0000 7fc2d68ed500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.455+0000 7fc2d68ed500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:57.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.455+0000 7fc2d68ed500 20 get_system_obj_state: rctx=0x7fff9aeb2a20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55eb94e7b810 s->prefetch_data=0 2023-09-22T13:17:57.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.456+0000 7fc2d68ed500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.456+0000 7fc2d68ed500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.456+0000 7fc2d68ed500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.456+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.456+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.456+0000 7fc2d68ed500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.456+0000 7fc2d68ed500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:57.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.457+0000 7fc2d68ed500 20 get_system_obj_state: rctx=0x7fff9aeb2a20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55eb94e7b810 s->prefetch_data=0 2023-09-22T13:17:57.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.457+0000 7fc2d68ed500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.457+0000 7fc2d68ed500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.457+0000 7fc2d68ed500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.457+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.458+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.458+0000 7fc2d68ed500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.458+0000 7fc2d68ed500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:57.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.458+0000 7fc2d68ed500 20 get_system_obj_state: rctx=0x7fff9aeb2a20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55eb94e7b810 s->prefetch_data=0 2023-09-22T13:17:57.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.459+0000 7fc2d68ed500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.459+0000 7fc2d68ed500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.459+0000 7fc2d68ed500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.459+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.459+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.459+0000 7fc2d68ed500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.459+0000 7fc2d68ed500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:57.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.459+0000 7fc2d68ed500 20 get_system_obj_state: rctx=0x7fff9aeb2a20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55eb94e7b810 s->prefetch_data=0 2023-09-22T13:17:57.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.460+0000 7fc2d68ed500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.460+0000 7fc2d68ed500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.460+0000 7fc2d68ed500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.460+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.460+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.460+0000 7fc2d68ed500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.460+0000 7fc2d68ed500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:57.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.461+0000 7fc2d68ed500 20 get_system_obj_state: rctx=0x7fff9aeb2a20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55eb94e7b810 s->prefetch_data=0 2023-09-22T13:17:57.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.461+0000 7fc2d68ed500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.461+0000 7fc2d68ed500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.461+0000 7fc2d68ed500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.461+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.461+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.461+0000 7fc2d68ed500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.461+0000 7fc2d68ed500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:57.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.462+0000 7fc2d68ed500 20 get_system_obj_state: rctx=0x7fff9aeb2a20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55eb94e7b810 s->prefetch_data=0 2023-09-22T13:17:57.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.462+0000 7fc2d68ed500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.462+0000 7fc2d68ed500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.462+0000 7fc2d68ed500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.462+0000 7fc2d68ed500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.462+0000 7fc2d68ed500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.462+0000 7fc2d68ed500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.462+0000 7fc2d68ed500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:57.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.464+0000 7fc2d68ed500 20 remove_watcher() i=0 2023-09-22T13:17:57.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.464+0000 7fc2d68ed500 2 removed watcher, disabling cache 2023-09-22T13:17:57.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.465+0000 7fc2d68ed500 20 remove_watcher() i=1 2023-09-22T13:17:57.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.466+0000 7fc2d68ed500 20 remove_watcher() i=2 2023-09-22T13:17:57.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.467+0000 7fc2d68ed500 20 remove_watcher() i=3 2023-09-22T13:17:57.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.467+0000 7fc2d68ed500 20 remove_watcher() i=4 2023-09-22T13:17:57.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.468+0000 7fc2d68ed500 20 remove_watcher() i=5 2023-09-22T13:17:57.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.469+0000 7fc2d68ed500 20 remove_watcher() i=6 2023-09-22T13:17:57.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.470+0000 7fc2d68ed500 20 remove_watcher() i=7 2023-09-22T13:17:57.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.470+0000 7fc292fe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:57.661 INFO:tasks.workunit.client.0.smithi096.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-09-22T13:17:57.661 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bi list --bucket check-bucket --object e 2023-09-22T13:17:57.661 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:57.661 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:57.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.563+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.565+0000 7f6eb1829500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.565+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.566+0000 7f6eb1829500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.566+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.566+0000 7f6eb1829500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.566+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.567+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.576+0000 7f6eb1829500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:57.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.576+0000 7f6eb1829500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:57.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.576+0000 7f6eb1829500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:57.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.576+0000 7f6eb1829500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:57.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.576+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.663 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.577+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.663 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.577+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.663 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.577+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:57.663 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.577+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.663 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.577+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.663 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.577+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.663 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.578+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:57.663 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.578+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.663 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.578+0000 7f6eb1829500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.663 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.578+0000 7f6eb1829500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:57.663 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.578+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.663 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.578+0000 7f6eb1829500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.663 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.578+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.664 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.579+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.664 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.579+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.664 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.579+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:57.664 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.579+0000 7f6eb1829500 20 zonegroup default 2023-09-22T13:17:57.664 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.579+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.664 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.579+0000 7f6eb1829500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.664 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.579+0000 7f6eb1829500 10 Cannot find current period zone using local zone 2023-09-22T13:17:57.664 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.579+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.664 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.579+0000 7f6eb1829500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.664 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.579+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.664 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.580+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.664 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.580+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.664 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.580+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:57.665 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.580+0000 7f6eb1829500 20 zone default found 2023-09-22T13:17:57.665 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.580+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.665 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.580+0000 7f6eb1829500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.665 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.580+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.665 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.581+0000 7f6eb1829500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.665 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.581+0000 7f6eb1829500 20 started sync module instance, tier type = 2023-09-22T13:17:57.665 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.581+0000 7f6eb1829500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:57.665 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.587+0000 7f6eb1829500 20 add_watcher() i=0 2023-09-22T13:17:57.665 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.587+0000 7f6eb1829500 20 add_watcher() i=1 2023-09-22T13:17:57.665 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.587+0000 7f6eb1829500 20 add_watcher() i=2 2023-09-22T13:17:57.665 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.587+0000 7f6eb1829500 20 add_watcher() i=3 2023-09-22T13:17:57.665 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.587+0000 7f6eb1829500 20 add_watcher() i=4 2023-09-22T13:17:57.666 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.587+0000 7f6eb1829500 20 add_watcher() i=5 2023-09-22T13:17:57.666 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.587+0000 7f6eb1829500 20 add_watcher() i=6 2023-09-22T13:17:57.666 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.588+0000 7f6eb1829500 20 add_watcher() i=7 2023-09-22T13:17:57.666 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.588+0000 7f6eb1829500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:57.666 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.588+0000 7f6e9d50c700 0 -- 172.21.15.96:0/3224546352 >> [v2:172.21.15.96:6816/140810,v1:172.21.15.96:6817/140810] conn(0x5563bcfd0420 msgr2=0x5563bcfd2810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-09-22T13:17:57.666 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.590+0000 7f6e6dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:57.666 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.590+0000 7f6eb1829500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:57.666 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.590+0000 7f6eb1829500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:57.666 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.594+0000 7f6e9cd0b700 0 -- 172.21.15.96:0/3921128800 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5563bd0173c0 msgr2=0x5563bd026a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-09-22T13:17:57.666 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.597+0000 7f6eb1829500 5 note: GC not initialized 2023-09-22T13:17:57.666 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.598+0000 7f6de4ff9700 20 reqs_thread_entry: start 2023-09-22T13:17:57.666 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.598+0000 7f6dd27fc700 20 reqs_thread_entry: start 2023-09-22T13:17:57.667 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.598+0000 7f6eb1829500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:57.667 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.598+0000 7f6eb1829500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:57.667 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.598+0000 7f6dd0ff9700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:21 2023 2023-09-22T13:17:57.667 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:57.667 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.598+0000 7f6dbbfff700 20 reqs_thread_entry: start 2023-09-22T13:17:57.667 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.598+0000 7f6eb1829500 20 get_system_obj_state: rctx=0x7ffff2e873c0 obj=default.rgw.meta:root:check-bucket state=0x5563bd018480 s->prefetch_data=0 2023-09-22T13:17:57.667 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.598+0000 7f6eb1829500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.667 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.598+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.667 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.599+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:57.667 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.599+0000 7f6eb1829500 20 get_system_obj_state: rctx=0x7ffff2e873c0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5563bcfeb220 s->prefetch_data=0 2023-09-22T13:17:57.667 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.599+0000 7f6eb1829500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.667 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.599+0000 7f6eb1829500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.668 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.599+0000 7f6eb1829500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.668 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.599+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.668 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.599+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.668 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.599+0000 7f6eb1829500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.668 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.599+0000 7f6eb1829500 20 get_system_obj_state: rctx=0x7ffff2e87190 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5563bcfeb220 s->prefetch_data=0 2023-09-22T13:17:57.668 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.600+0000 7f6eb1829500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.668 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.600+0000 7f6eb1829500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.668 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.600+0000 7f6eb1829500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.668 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.600+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.668 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.600+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.668 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.600+0000 7f6eb1829500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.668 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.600+0000 7f6eb1829500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:57.668 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.601+0000 7f6eb1829500 20 get_system_obj_state: rctx=0x7ffff2e87190 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5563bcfeb220 s->prefetch_data=0 2023-09-22T13:17:57.668 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.601+0000 7f6eb1829500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.669 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.601+0000 7f6eb1829500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.669 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.601+0000 7f6eb1829500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.669 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.601+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.910 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.601+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.910 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.601+0000 7f6eb1829500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.910 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.601+0000 7f6eb1829500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:57.911 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.602+0000 7f6eb1829500 20 get_system_obj_state: rctx=0x7ffff2e87190 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5563bcfeb220 s->prefetch_data=0 2023-09-22T13:17:57.911 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.602+0000 7f6eb1829500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.911 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.602+0000 7f6eb1829500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.911 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.602+0000 7f6eb1829500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.911 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.602+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.911 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.603+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.911 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.603+0000 7f6eb1829500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.911 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.603+0000 7f6eb1829500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:57.911 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.603+0000 7f6eb1829500 20 get_system_obj_state: rctx=0x7ffff2e87190 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5563bcfeb220 s->prefetch_data=0 2023-09-22T13:17:57.911 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.604+0000 7f6eb1829500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.911 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.604+0000 7f6eb1829500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.911 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.604+0000 7f6eb1829500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.912 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.604+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.912 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.604+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.912 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.604+0000 7f6eb1829500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.912 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.604+0000 7f6eb1829500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:57.912 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.604+0000 7f6eb1829500 20 get_system_obj_state: rctx=0x7ffff2e87190 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5563bcfeb220 s->prefetch_data=0 2023-09-22T13:17:57.912 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.605+0000 7f6eb1829500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.912 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.605+0000 7f6eb1829500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.912 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.605+0000 7f6eb1829500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.912 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.605+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.912 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.605+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.912 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.605+0000 7f6eb1829500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.912 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.605+0000 7f6eb1829500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:57.913 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.605+0000 7f6eb1829500 20 get_system_obj_state: rctx=0x7ffff2e87190 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5563bcfeb220 s->prefetch_data=0 2023-09-22T13:17:57.913 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.606+0000 7f6eb1829500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.913 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.606+0000 7f6eb1829500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.913 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.606+0000 7f6eb1829500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.913 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.606+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.913 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.606+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.913 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.606+0000 7f6eb1829500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.913 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.606+0000 7f6eb1829500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:57.913 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.607+0000 7f6eb1829500 20 get_system_obj_state: rctx=0x7ffff2e87190 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5563bcfeb220 s->prefetch_data=0 2023-09-22T13:17:57.913 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.607+0000 7f6eb1829500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.913 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.607+0000 7f6eb1829500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.913 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.607+0000 7f6eb1829500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.914 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.607+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.914 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.607+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.914 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.607+0000 7f6eb1829500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.914 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.607+0000 7f6eb1829500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:57.914 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.608+0000 7f6eb1829500 20 get_system_obj_state: rctx=0x7ffff2e87190 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5563bcfeb220 s->prefetch_data=0 2023-09-22T13:17:57.914 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.608+0000 7f6eb1829500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.914 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.608+0000 7f6eb1829500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.914 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.608+0000 7f6eb1829500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.914 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.608+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.914 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.609+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.914 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.609+0000 7f6eb1829500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.914 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.609+0000 7f6eb1829500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:57.915 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.609+0000 7f6eb1829500 20 get_system_obj_state: rctx=0x7ffff2e87190 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5563bcfeb220 s->prefetch_data=0 2023-09-22T13:17:57.915 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.609+0000 7f6eb1829500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.915 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.609+0000 7f6eb1829500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.915 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.609+0000 7f6eb1829500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.915 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.609+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.915 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.610+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.915 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.610+0000 7f6eb1829500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.915 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.610+0000 7f6eb1829500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:57.915 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.610+0000 7f6eb1829500 20 get_system_obj_state: rctx=0x7ffff2e87190 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5563bcfeb220 s->prefetch_data=0 2023-09-22T13:17:57.915 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.611+0000 7f6eb1829500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.915 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.611+0000 7f6eb1829500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.916 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.611+0000 7f6eb1829500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.916 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.611+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.916 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.611+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.916 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.611+0000 7f6eb1829500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.916 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.611+0000 7f6eb1829500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:57.916 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.611+0000 7f6eb1829500 20 get_system_obj_state: rctx=0x7ffff2e87190 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5563bcfeb220 s->prefetch_data=0 2023-09-22T13:17:57.916 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.612+0000 7f6eb1829500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.916 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.612+0000 7f6eb1829500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.916 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.612+0000 7f6eb1829500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.916 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.612+0000 7f6eb1829500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.916 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.612+0000 7f6eb1829500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.916 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.612+0000 7f6eb1829500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.917 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.612+0000 7f6eb1829500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:57.917 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.614+0000 7f6eb1829500 20 remove_watcher() i=0 2023-09-22T13:17:57.917 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.614+0000 7f6eb1829500 2 removed watcher, disabling cache 2023-09-22T13:17:57.917 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.614+0000 7f6eb1829500 20 remove_watcher() i=1 2023-09-22T13:17:57.917 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.615+0000 7f6eb1829500 20 remove_watcher() i=2 2023-09-22T13:17:57.917 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.616+0000 7f6eb1829500 20 remove_watcher() i=3 2023-09-22T13:17:57.917 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.617+0000 7f6eb1829500 20 remove_watcher() i=4 2023-09-22T13:17:57.917 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.618+0000 7f6eb1829500 20 remove_watcher() i=5 2023-09-22T13:17:57.917 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.618+0000 7f6eb1829500 20 remove_watcher() i=6 2023-09-22T13:17:57.917 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.619+0000 7f6eb1829500 20 remove_watcher() i=7 2023-09-22T13:17:57.917 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.620+0000 7f6e6dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:57.917 INFO:tasks.workunit.client.0.smithi096.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-09-22T13:17:57.918 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bi list --bucket check-bucket --object f 2023-09-22T13:17:57.918 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:57.918 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:57.918 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.709+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.918 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.711+0000 7fd3f5338500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.918 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.711+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.918 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.712+0000 7fd3f5338500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.918 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.712+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.918 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.712+0000 7fd3f5338500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.918 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.712+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.918 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.713+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.918 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.722+0000 7fd3f5338500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:57.919 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.722+0000 7fd3f5338500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:57.919 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.722+0000 7fd3f5338500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:57.919 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.722+0000 7fd3f5338500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:57.919 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.722+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.919 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.723+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.919 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.723+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.919 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.723+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:57.919 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.723+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.919 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.723+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.919 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.723+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.919 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.723+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:57.920 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.724+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.920 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.724+0000 7fd3f5338500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.920 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.724+0000 7fd3f5338500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:57.920 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.724+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.920 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.724+0000 7fd3f5338500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.920 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.724+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.920 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.724+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.920 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.724+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.920 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.724+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:57.920 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.724+0000 7fd3f5338500 20 zonegroup default 2023-09-22T13:17:57.920 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.724+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.921 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.725+0000 7fd3f5338500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.921 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.725+0000 7fd3f5338500 10 Cannot find current period zone using local zone 2023-09-22T13:17:57.921 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.725+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.921 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.725+0000 7fd3f5338500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.921 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.725+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.921 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.725+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.921 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.725+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.921 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.725+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:57.921 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.725+0000 7fd3f5338500 20 zone default found 2023-09-22T13:17:57.921 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.725+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.921 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.726+0000 7fd3f5338500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.922 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.726+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.922 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.726+0000 7fd3f5338500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.922 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.726+0000 7fd3f5338500 20 started sync module instance, tier type = 2023-09-22T13:17:57.922 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.726+0000 7fd3f5338500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:57.922 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.733+0000 7fd3f5338500 20 add_watcher() i=0 2023-09-22T13:17:57.922 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.733+0000 7fd3f5338500 20 add_watcher() i=1 2023-09-22T13:17:57.922 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.733+0000 7fd3f5338500 20 add_watcher() i=2 2023-09-22T13:17:57.922 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.733+0000 7fd3f5338500 20 add_watcher() i=3 2023-09-22T13:17:57.922 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.733+0000 7fd3f5338500 20 add_watcher() i=4 2023-09-22T13:17:57.922 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.733+0000 7fd3f5338500 20 add_watcher() i=5 2023-09-22T13:17:57.923 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.733+0000 7fd3f5338500 20 add_watcher() i=6 2023-09-22T13:17:57.923 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.733+0000 7fd3e081a700 0 -- 172.21.15.96:0/1372578778 >> [v2:172.21.15.96:6807/140797,v1:172.21.15.96:6809/140797] conn(0x7fd3d401c8d0 msgr2=0x7fd3d400d3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:57.923 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.733+0000 7fd3f5338500 20 add_watcher() i=7 2023-09-22T13:17:57.923 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.733+0000 7fd3f5338500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:57.923 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.735+0000 7fd3b1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:57.923 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.735+0000 7fd3f5338500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:57.923 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.735+0000 7fd3f5338500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:57.923 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.739+0000 7fd3e101b700 0 -- 172.21.15.96:0/930386373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556f57c5b1a0 msgr2=0x556f57c68300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:57.923 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.940+0000 7fd3e101b700 0 --2- 172.21.15.96:0/930386373 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556f57c5b1a0 0x556f57c68300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-09-22T13:17:57.923 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.942+0000 7fd3f5338500 5 note: GC not initialized 2023-09-22T13:17:57.923 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.943+0000 7fd328ff9700 20 reqs_thread_entry: start 2023-09-22T13:17:57.923 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.943+0000 7fd31e7fc700 20 reqs_thread_entry: start 2023-09-22T13:17:57.924 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.943+0000 7fd3f5338500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:57.924 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.943+0000 7fd3f5338500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:57.924 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.943+0000 7fd31cff9700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:22 2023 2023-09-22T13:17:57.924 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:57.924 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.943+0000 7fd303fff700 20 reqs_thread_entry: start 2023-09-22T13:17:57.924 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.943+0000 7fd3f5338500 20 get_system_obj_state: rctx=0x7ffeb6fb76e0 obj=default.rgw.meta:root:check-bucket state=0x556f57c60680 s->prefetch_data=0 2023-09-22T13:17:57.924 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.944+0000 7fd3f5338500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.924 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.944+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.924 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.944+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:57.924 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.944+0000 7fd3f5338500 20 get_system_obj_state: rctx=0x7ffeb6fb76e0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x556f57aae810 s->prefetch_data=0 2023-09-22T13:17:57.924 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.944+0000 7fd3f5338500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.925 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.944+0000 7fd3f5338500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.925 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.944+0000 7fd3f5338500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.925 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.944+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.925 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.945+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.925 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.945+0000 7fd3f5338500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.925 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.945+0000 7fd3f5338500 20 get_system_obj_state: rctx=0x7ffeb6fb74b0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x556f57aae810 s->prefetch_data=0 2023-09-22T13:17:57.925 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.945+0000 7fd3f5338500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.925 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.945+0000 7fd3f5338500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.925 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.945+0000 7fd3f5338500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.925 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.945+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.925 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.945+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.925 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.945+0000 7fd3f5338500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.926 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.945+0000 7fd3f5338500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:57.926 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.946+0000 7fd3f5338500 20 get_system_obj_state: rctx=0x7ffeb6fb74b0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x556f57aae810 s->prefetch_data=0 2023-09-22T13:17:57.926 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.946+0000 7fd3f5338500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.926 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.946+0000 7fd3f5338500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.926 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.946+0000 7fd3f5338500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.926 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.946+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.926 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.947+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.926 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.947+0000 7fd3f5338500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.926 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.947+0000 7fd3f5338500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:57.926 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.947+0000 7fd3f5338500 20 get_system_obj_state: rctx=0x7ffeb6fb74b0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x556f57aae810 s->prefetch_data=0 2023-09-22T13:17:57.926 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.947+0000 7fd3f5338500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.927 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.947+0000 7fd3f5338500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.927 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.947+0000 7fd3f5338500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.927 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.947+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.927 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.948+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.927 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.948+0000 7fd3f5338500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.927 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.948+0000 7fd3f5338500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:57.927 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.948+0000 7fd3f5338500 20 get_system_obj_state: rctx=0x7ffeb6fb74b0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x556f57aae810 s->prefetch_data=0 2023-09-22T13:17:57.927 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.949+0000 7fd3f5338500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.927 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.949+0000 7fd3f5338500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.927 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.949+0000 7fd3f5338500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.927 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.949+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.927 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.949+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.928 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.949+0000 7fd3f5338500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.928 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.949+0000 7fd3f5338500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:57.928 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.949+0000 7fd3f5338500 20 get_system_obj_state: rctx=0x7ffeb6fb74b0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x556f57aae810 s->prefetch_data=0 2023-09-22T13:17:57.928 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.950+0000 7fd3f5338500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.928 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.950+0000 7fd3f5338500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.928 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.950+0000 7fd3f5338500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.928 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.950+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.928 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.950+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.928 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.950+0000 7fd3f5338500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.928 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.950+0000 7fd3f5338500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:57.928 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.951+0000 7fd3f5338500 20 get_system_obj_state: rctx=0x7ffeb6fb74b0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x556f57aae810 s->prefetch_data=0 2023-09-22T13:17:57.928 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.951+0000 7fd3f5338500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.929 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.951+0000 7fd3f5338500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.929 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.951+0000 7fd3f5338500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.929 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.951+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.929 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.951+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.929 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.951+0000 7fd3f5338500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.929 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.951+0000 7fd3f5338500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:57.929 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.952+0000 7fd3f5338500 20 get_system_obj_state: rctx=0x7ffeb6fb74b0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x556f57aae810 s->prefetch_data=0 2023-09-22T13:17:57.929 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.952+0000 7fd3f5338500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.929 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.952+0000 7fd3f5338500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.929 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.952+0000 7fd3f5338500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.929 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.952+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.929 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.952+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.929 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.952+0000 7fd3f5338500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.930 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.952+0000 7fd3f5338500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:57.930 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.953+0000 7fd3f5338500 20 get_system_obj_state: rctx=0x7ffeb6fb74b0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x556f57aae810 s->prefetch_data=0 2023-09-22T13:17:57.930 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.954+0000 7fd3f5338500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.930 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.954+0000 7fd3f5338500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.930 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.954+0000 7fd3f5338500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.930 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.954+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.930 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.954+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.930 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.954+0000 7fd3f5338500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.930 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.954+0000 7fd3f5338500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:57.930 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.954+0000 7fd3f5338500 20 get_system_obj_state: rctx=0x7ffeb6fb74b0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x556f57aae810 s->prefetch_data=0 2023-09-22T13:17:57.930 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.955+0000 7fd3f5338500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.931 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.955+0000 7fd3f5338500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.931 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.955+0000 7fd3f5338500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.931 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.955+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.931 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.955+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.931 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.955+0000 7fd3f5338500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.931 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.955+0000 7fd3f5338500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:57.931 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.956+0000 7fd3f5338500 20 get_system_obj_state: rctx=0x7ffeb6fb74b0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x556f57aae810 s->prefetch_data=0 2023-09-22T13:17:57.931 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.956+0000 7fd3f5338500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.931 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.956+0000 7fd3f5338500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.931 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.956+0000 7fd3f5338500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.931 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.956+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.931 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.956+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.932 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.956+0000 7fd3f5338500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.932 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.956+0000 7fd3f5338500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:57.932 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.957+0000 7fd3f5338500 20 get_system_obj_state: rctx=0x7ffeb6fb74b0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x556f57aae810 s->prefetch_data=0 2023-09-22T13:17:57.932 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.957+0000 7fd3f5338500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:57.932 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.957+0000 7fd3f5338500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:57.932 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.957+0000 7fd3f5338500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:57.932 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.957+0000 7fd3f5338500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.932 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.957+0000 7fd3f5338500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:57.932 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.957+0000 7fd3f5338500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:57.932 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.957+0000 7fd3f5338500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:57.932 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.959+0000 7fd3f5338500 20 remove_watcher() i=0 2023-09-22T13:17:57.932 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.959+0000 7fd3f5338500 2 removed watcher, disabling cache 2023-09-22T13:17:57.933 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.960+0000 7fd3f5338500 20 remove_watcher() i=1 2023-09-22T13:17:57.933 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.960+0000 7fd3f5338500 20 remove_watcher() i=2 2023-09-22T13:17:57.933 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.961+0000 7fd3f5338500 20 remove_watcher() i=3 2023-09-22T13:17:57.933 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.962+0000 7fd3f5338500 20 remove_watcher() i=4 2023-09-22T13:17:57.933 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.963+0000 7fd3f5338500 20 remove_watcher() i=5 2023-09-22T13:17:57.933 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.963+0000 7fd3f5338500 20 remove_watcher() i=6 2023-09-22T13:17:57.933 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.964+0000 7fd3f5338500 20 remove_watcher() i=7 2023-09-22T13:17:57.933 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:51.965+0000 7fd3b1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:57.933 INFO:tasks.workunit.client.0.smithi096.stderr:ceph config rm client rgw_debug_inject_set_olh_err 2023-09-22T13:17:57.933 INFO:tasks.workunit.client.0.smithi096.stderr:ceph config rm client rgw_debug_inject_olh_cancel_modification_err 2023-09-22T13:17:57.933 INFO:tasks.workunit.client.0.smithi096.stderr:TEST: bucket check unlinked finds unlistable entries 2023-09-22T13:17:57.933 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:57.934 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --min-age-hours 0 --dump-keys 2023-09-22T13:17:57.934 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:57.934 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:57.934 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.788+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.934 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.790+0000 7f161a8eb500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.934 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.790+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.934 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.790+0000 7f161a8eb500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.934 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.790+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.934 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.791+0000 7f161a8eb500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:57.934 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.791+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.934 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.791+0000 7f161a8eb500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.935 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.801+0000 7f161a8eb500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:57.935 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.801+0000 7f161a8eb500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:57.935 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.801+0000 7f161a8eb500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:57.935 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.801+0000 7f161a8eb500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:57.935 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.801+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.935 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.802+0000 7f161a8eb500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.935 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.802+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.935 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.802+0000 7f161a8eb500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:57.935 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.802+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.935 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.802+0000 7f161a8eb500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:57.935 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.802+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:57.935 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.803+0000 7f161a8eb500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:57.936 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.803+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.101 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.803+0000 7f161a8eb500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.102 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.803+0000 7f161a8eb500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:58.102 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.803+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.102 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.803+0000 7f161a8eb500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.102 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.803+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.102 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.803+0000 7f161a8eb500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.102 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.803+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.102 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.804+0000 7f161a8eb500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.102 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.804+0000 7f161a8eb500 20 zonegroup default 2023-09-22T13:17:58.102 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.804+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.102 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.804+0000 7f161a8eb500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.103 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.804+0000 7f161a8eb500 10 Cannot find current period zone using local zone 2023-09-22T13:17:58.103 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.804+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.103 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.804+0000 7f161a8eb500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.103 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.804+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.103 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.804+0000 7f161a8eb500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.103 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.804+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.103 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.805+0000 7f161a8eb500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.103 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.805+0000 7f161a8eb500 20 zone default found 2023-09-22T13:17:58.103 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.805+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.103 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.805+0000 7f161a8eb500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.103 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.805+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.103 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.805+0000 7f161a8eb500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.103 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.805+0000 7f161a8eb500 20 started sync module instance, tier type = 2023-09-22T13:17:58.104 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.805+0000 7f161a8eb500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:58.104 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.812+0000 7f161a8eb500 20 add_watcher() i=0 2023-09-22T13:17:58.104 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.812+0000 7f161a8eb500 20 add_watcher() i=1 2023-09-22T13:17:58.104 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.812+0000 7f161a8eb500 20 add_watcher() i=2 2023-09-22T13:17:58.104 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.812+0000 7f161a8eb500 20 add_watcher() i=3 2023-09-22T13:17:58.104 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.812+0000 7f161a8eb500 20 add_watcher() i=4 2023-09-22T13:17:58.104 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.812+0000 7f161a8eb500 20 add_watcher() i=5 2023-09-22T13:17:58.104 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.812+0000 7f161a8eb500 20 add_watcher() i=6 2023-09-22T13:17:58.104 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.813+0000 7f161a8eb500 20 add_watcher() i=7 2023-09-22T13:17:58.104 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.813+0000 7f161a8eb500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:58.104 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.814+0000 7f16065ce700 0 -- 172.21.15.96:0/264574833 >> [v2:172.21.15.96:6816/140810,v1:172.21.15.96:6817/140810] conn(0x55dff65924c0 msgr2=0x55dff65948b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:58.104 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.814+0000 7f15d77de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.105 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.815+0000 7f161a8eb500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:58.105 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.815+0000 7f161a8eb500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:58.105 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.818+0000 7f16065ce700 0 -- 172.21.15.96:0/4263281858 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55dff65931b0 msgr2=0x55dff65d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:58.105 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.820+0000 7f161a8eb500 5 note: GC not initialized 2023-09-22T13:17:58.105 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.821+0000 7f15467fc700 20 reqs_thread_entry: start 2023-09-22T13:17:58.105 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.822+0000 7f1537fff700 20 reqs_thread_entry: start 2023-09-22T13:17:58.105 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.822+0000 7f161a8eb500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:58.105 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.822+0000 7f161a8eb500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:58.105 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.822+0000 7f15367fc700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:23 2023 2023-09-22T13:17:58.105 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:58.105 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.822+0000 7f1535ffb700 20 reqs_thread_entry: start 2023-09-22T13:17:58.106 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.822+0000 7f161a8eb500 20 get_system_obj_state: rctx=0x7ffc4a803888 obj=default.rgw.meta:root:check-bucket state=0x55dff655b690 s->prefetch_data=0 2023-09-22T13:17:58.106 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.822+0000 7f161a8eb500 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-09-22T13:17:58.106 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.822+0000 7f161a8eb500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x16 2023-09-22T13:17:58.106 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.822+0000 7f161a8eb500 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:58.106 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.822+0000 7f161a8eb500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:58.106 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.822+0000 7f161a8eb500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.106 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.822+0000 7f161a8eb500 10 cache get: name=default.rgw.meta+root+check-bucket : type miss (requested=0x11, cached=0x16) 2023-09-22T13:17:58.106 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.822+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.106 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:58.106 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-09-22T13:17:58.106 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 10 moving default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:58.106 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 20 get_system_obj_state: rctx=0x7ffc4a803958 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55dff655b690 s->prefetch_data=0 2023-09-22T13:17:58.107 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : miss 2023-09-22T13:17:58.107 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x16 2023-09-22T13:17:58.107 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:58.107 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:58.107 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:58.107 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.107 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.107 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.107 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : type miss (requested=0x13, cached=0x16) 2023-09-22T13:17:58.107 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.107 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.107 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x13 2023-09-22T13:17:58.107 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 10 moving default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:58.108 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:58.108 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:58.108 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 2023-09-22T13:17:58.108 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.823+0000 7f161a8eb500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:58.108 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.824+0000 7f161a8eb500 1 NOTICE: finished shard 0 (0 entries found) 2023-09-22T13:17:58.108 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.824+0000 7f161a8eb500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:58.108 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.825+0000 7f161a8eb500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:58.108 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.825+0000 7f161a8eb500 1 NOTICE: finished shard 2 (0 entries found) 2023-09-22T13:17:58.108 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.825+0000 7f161a8eb500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:58.108 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.826+0000 7f161a8eb500 1 NOTICE: finished shard 3 (0 entries found) 2023-09-22T13:17:58.108 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.826+0000 7f161a8eb500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:58.108 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.826+0000 7f161a8eb500 1 NOTICE: finished shard 4 (0 entries found) 2023-09-22T13:17:58.109 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.826+0000 7f161a8eb500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.109 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.827+0000 7f161a8eb500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:58.109 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.828+0000 7f161a8eb500 1 NOTICE: finished shard 6 (0 entries found) 2023-09-22T13:17:58.109 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.828+0000 7f161a8eb500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:58.109 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.828+0000 7f161a8eb500 1 NOTICE: finished shard 7 (0 entries found) 2023-09-22T13:17:58.109 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.828+0000 7f161a8eb500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.109 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.829+0000 7f161a8eb500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:58.109 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.830+0000 7f161a8eb500 1 NOTICE: finished shard 9 (0 entries found) 2023-09-22T13:17:58.109 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.830+0000 7f161a8eb500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:58.109 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.830+0000 7f161a8eb500 1 NOTICE: finished shard 1 (1 entries found) 2023-09-22T13:17:58.109 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.830+0000 7f161a8eb500 1 NOTICE: finished shard 5 (1 entries found) 2023-09-22T13:17:58.109 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.830+0000 7f161a8eb500 1 NOTICE: finished shard 8 (1 entries found) 2023-09-22T13:17:58.109 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.830+0000 7f161a8eb500 1 NOTICE: finished shard 10 (1 entries found) 2023-09-22T13:17:58.110 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.830+0000 7f161a8eb500 1 NOTICE: finished all shards (4 entries found) 2023-09-22T13:17:58.110 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.832+0000 7f161a8eb500 20 remove_watcher() i=0 2023-09-22T13:17:58.110 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.832+0000 7f161a8eb500 2 removed watcher, disabling cache 2023-09-22T13:17:58.110 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.832+0000 7f161a8eb500 20 remove_watcher() i=1 2023-09-22T13:17:58.110 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.833+0000 7f161a8eb500 20 remove_watcher() i=2 2023-09-22T13:17:58.110 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.834+0000 7f161a8eb500 20 remove_watcher() i=3 2023-09-22T13:17:58.110 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.835+0000 7f161a8eb500 20 remove_watcher() i=4 2023-09-22T13:17:58.110 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.836+0000 7f161a8eb500 20 remove_watcher() i=5 2023-09-22T13:17:58.110 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.836+0000 7f161a8eb500 20 remove_watcher() i=6 2023-09-22T13:17:58.110 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.837+0000 7f161a8eb500 20 remove_watcher() i=7 2023-09-22T13:17:58.110 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.838+0000 7f15d77de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.110 INFO:tasks.workunit.client.0.smithi096.stderr:TEST: unlinked entries are not listable 2023-09-22T13:17:58.111 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:58.111 INFO:tasks.workunit.client.0.smithi096.stderr:TEST: GET returns 404 for unlinked entry keys that have no other versions 2023-09-22T13:17:58.111 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:58.111 INFO:tasks.workunit.client.0.smithi096.stderr:TEST: bucket check unlinked fixes unlistable entries 2023-09-22T13:17:58.111 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:58.111 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --fix --min-age-hours 0 --rgw-olh-pending-timeout-sec 0 --dump-keys 2023-09-22T13:17:58.111 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:58.111 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:58.111 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.977+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.111 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.978+0000 7fd98c0da500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.111 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.978+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.111 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.979+0000 7fd98c0da500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.111 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.979+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.112 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.979+0000 7fd98c0da500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.112 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.979+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.112 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.980+0000 7fd98c0da500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.112 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.991+0000 7fd98c0da500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:58.112 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.991+0000 7fd98c0da500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:58.112 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.991+0000 7fd98c0da500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:58.112 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.991+0000 7fd98c0da500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:58.112 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.991+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.112 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.991+0000 7fd98c0da500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.112 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.991+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.112 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.991+0000 7fd98c0da500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.112 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.991+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.112 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.992+0000 7fd98c0da500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.113 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.992+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.113 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.992+0000 7fd98c0da500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.113 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.992+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.113 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.992+0000 7fd98c0da500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.113 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.992+0000 7fd98c0da500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:58.113 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.992+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.113 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.992+0000 7fd98c0da500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.113 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.992+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.113 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.992+0000 7fd98c0da500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.113 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.992+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.113 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.993+0000 7fd98c0da500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.113 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.993+0000 7fd98c0da500 20 zonegroup default 2023-09-22T13:17:58.113 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.993+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.114 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.993+0000 7fd98c0da500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.114 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.993+0000 7fd98c0da500 10 Cannot find current period zone using local zone 2023-09-22T13:17:58.114 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.993+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.114 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.993+0000 7fd98c0da500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.114 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.993+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.114 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.993+0000 7fd98c0da500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.114 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.993+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.114 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.994+0000 7fd98c0da500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.114 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.994+0000 7fd98c0da500 20 zone default found 2023-09-22T13:17:58.114 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.994+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.114 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.994+0000 7fd98c0da500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.114 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.994+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.115 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.994+0000 7fd98c0da500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.115 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.994+0000 7fd98c0da500 20 started sync module instance, tier type = 2023-09-22T13:17:58.115 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:52.994+0000 7fd98c0da500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:58.115 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.001+0000 7fd98c0da500 20 add_watcher() i=0 2023-09-22T13:17:58.115 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.001+0000 7fd98c0da500 20 add_watcher() i=1 2023-09-22T13:17:58.115 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.001+0000 7fd98c0da500 20 add_watcher() i=2 2023-09-22T13:17:58.115 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.001+0000 7fd98c0da500 20 add_watcher() i=3 2023-09-22T13:17:58.115 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.001+0000 7fd98c0da500 20 add_watcher() i=4 2023-09-22T13:17:58.115 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.001+0000 7fd98c0da500 20 add_watcher() i=5 2023-09-22T13:17:58.115 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.001+0000 7fd98c0da500 20 add_watcher() i=6 2023-09-22T13:17:58.115 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.002+0000 7fd98c0da500 20 add_watcher() i=7 2023-09-22T13:17:58.115 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.002+0000 7fd98c0da500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:58.115 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.003+0000 7fd977dbd700 0 -- 172.21.15.96:0/3222522057 >> [v2:172.21.15.96:6816/140810,v1:172.21.15.96:6817/140810] conn(0x560fea1606a0 msgr2=0x560fea162a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:58.116 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.003+0000 7fd8c3fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.116 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.004+0000 7fd98c0da500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:58.116 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.004+0000 7fd98c0da500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:58.116 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.008+0000 7fd9775bc700 0 -- 172.21.15.96:0/2334584492 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x560fea1a8180 msgr2=0x560fea1b0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:58.116 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.010+0000 7fd98c0da500 5 note: GC not initialized 2023-09-22T13:17:58.116 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.011+0000 7fd8b2ffd700 20 reqs_thread_entry: start 2023-09-22T13:17:58.116 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.011+0000 7fd8b0ff9700 20 reqs_thread_entry: start 2023-09-22T13:17:58.116 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.011+0000 7fd98c0da500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:58.116 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.011+0000 7fd98c0da500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:58.116 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.012+0000 7fd8a2ffd700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:23 2023 2023-09-22T13:17:58.116 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:58.116 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.012+0000 7fd8a27fc700 20 reqs_thread_entry: start 2023-09-22T13:17:58.116 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.012+0000 7fd98c0da500 20 get_system_obj_state: rctx=0x7ffcae2b9738 obj=default.rgw.meta:root:check-bucket state=0x560fea0959e0 s->prefetch_data=0 2023-09-22T13:17:58.117 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.012+0000 7fd98c0da500 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-09-22T13:17:58.117 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.012+0000 7fd98c0da500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x16 2023-09-22T13:17:58.117 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.012+0000 7fd98c0da500 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:58.117 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.012+0000 7fd98c0da500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:58.117 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.012+0000 7fd98c0da500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.117 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.012+0000 7fd98c0da500 10 cache get: name=default.rgw.meta+root+check-bucket : type miss (requested=0x11, cached=0x16) 2023-09-22T13:17:58.117 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.012+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.117 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.012+0000 7fd98c0da500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:58.117 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.012+0000 7fd98c0da500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-09-22T13:17:58.117 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.012+0000 7fd98c0da500 10 moving default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:58.117 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.012+0000 7fd98c0da500 20 get_system_obj_state: rctx=0x7ffcae2b9808 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x560fea0959e0 s->prefetch_data=0 2023-09-22T13:17:58.117 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.012+0000 7fd98c0da500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : miss 2023-09-22T13:17:58.117 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.013+0000 7fd98c0da500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x16 2023-09-22T13:17:58.118 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.013+0000 7fd98c0da500 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:58.118 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.013+0000 7fd98c0da500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:58.118 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.013+0000 7fd98c0da500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:58.118 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.013+0000 7fd98c0da500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.118 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.013+0000 7fd98c0da500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.118 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.013+0000 7fd98c0da500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.118 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.013+0000 7fd98c0da500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : type miss (requested=0x13, cached=0x16) 2023-09-22T13:17:58.118 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.013+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.118 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.013+0000 7fd98c0da500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.118 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.013+0000 7fd98c0da500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x13 2023-09-22T13:17:58.118 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.013+0000 7fd98c0da500 10 moving default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:58.118 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.013+0000 7fd98c0da500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:58.119 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.013+0000 7fd98c0da500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:58.119 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.013+0000 7fd98c0da500 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 2023-09-22T13:17:58.119 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.013+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:58.119 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.014+0000 7fd98c0da500 1 NOTICE: finished shard 0 (0 entries removed) 2023-09-22T13:17:58.119 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.014+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:58.119 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.014+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:58.119 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.015+0000 7fd98c0da500 1 NOTICE: finished shard 2 (0 entries removed) 2023-09-22T13:17:58.119 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.015+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:58.119 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.015+0000 7fd98c0da500 1 NOTICE: finished shard 3 (0 entries removed) 2023-09-22T13:17:58.119 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.015+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:58.119 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.016+0000 7fd98c0da500 1 NOTICE: finished shard 4 (0 entries removed) 2023-09-22T13:17:58.119 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.016+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.120 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.017+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:58.120 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.017+0000 7fd98c0da500 1 NOTICE: finished shard 6 (0 entries removed) 2023-09-22T13:17:58.120 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.017+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:58.120 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.017+0000 7fd98c0da500 1 NOTICE: finished shard 7 (0 entries removed) 2023-09-22T13:17:58.120 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.017+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.120 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.018+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:58.120 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.018+0000 7fd98c0da500 1 NOTICE: finished shard 9 (0 entries removed) 2023-09-22T13:17:58.120 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.018+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:58.120 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.019+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:58.120 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.019+0000 7fd98c0da500 20 get_obj_state: rctx=0x560fea23f370 obj=check-bucket:e state=0x560fea285258 s->prefetch_data=0 2023-09-22T13:17:58.120 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.020+0000 7fd98c0da500 20 get_obj_state: setting s->obj_tag to qebj61b3pccx6sbsw90bayaity2d2121 2023-09-22T13:17:58.120 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.020+0000 7fd98c0da500 20 get_obj_state_impl: setting s->olh_tag to i411k1e0cez4bize043aabgvjc1i0i92 2023-09-22T13:17:58.120 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.020+0000 7fd98c0da500 20 bucket_index_guard_olh_op(): olh_state.olh_tag=i411k1e0cez4bize043aabgvjc1i0i92 2023-09-22T13:17:58.121 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.021+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:58.121 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.022+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:58.121 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.022+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:58.121 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.023+0000 7fd98c0da500 20 olh_log_entry: epoch=2 op=2 key=e[] 2023-09-22T13:17:58.121 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.023+0000 7fd98c0da500 20 olh_log_entry: epoch=2 op=3 key=e[6GTfBzVbkbXG8fIl1xFNb.KBgbcI26n] 2023-09-22T13:17:58.121 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.023+0000 7fd98c0da500 20 get_obj_state: rctx=0x560fea23f370 obj=check-bucket:_:6GTfBzVbkbXG8fIl1xFNb.KBgbcI26n_e state=0x560fea2ad308 s->prefetch_data=0 2023-09-22T13:17:58.121 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.023+0000 7fd98c0da500 10 manifest: total_size = 22 2023-09-22T13:17:58.121 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.023+0000 7fd98c0da500 20 get_obj_state: setting s->obj_tag to 42123edd-4a97-43ec-80bc-fc5350f88c11.4527.9502799470015272130 2023-09-22T13:17:58.121 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.023+0000 7fd98c0da500 20 get_obj_state: rctx=0x560fea23f370 obj=check-bucket:_:6GTfBzVbkbXG8fIl1xFNb.KBgbcI26n_e state=0x560fea2ad308 s->prefetch_data=0 2023-09-22T13:17:58.121 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.023+0000 7fd98c0da500 20 prepare_atomic_modification: state is not atomic. state=0x560fea2ad308 2023-09-22T13:17:58.121 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.023+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:58.121 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.026+0000 7fd98c0da500 10 cache get: name=default.rgw.log++bucket.sync-source-hints.check-bucket : miss 2023-09-22T13:17:58.121 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.026+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.122 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.026+0000 7fd98c0da500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.122 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.026+0000 7fd98c0da500 10 cache put: name=default.rgw.log++bucket.sync-source-hints.check-bucket info.flags=0x0 2023-09-22T13:17:58.122 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.026+0000 7fd98c0da500 10 adding default.rgw.log++bucket.sync-source-hints.check-bucket to cache LRU end 2023-09-22T13:17:58.122 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.026+0000 7fd98c0da500 10 cache get: name=default.rgw.log++bucket.sync-target-hints.check-bucket : miss 2023-09-22T13:17:58.122 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.026+0000 7fd98c0da500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.122 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.027+0000 7fd98c0da500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.122 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.027+0000 7fd98c0da500 10 cache put: name=default.rgw.log++bucket.sync-target-hints.check-bucket info.flags=0x0 2023-09-22T13:17:58.122 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.027+0000 7fd98c0da500 10 adding default.rgw.log++bucket.sync-target-hints.check-bucket to cache LRU end 2023-09-22T13:17:58.122 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.027+0000 7fd98c0da500 10 chain_cache_entry: cache_locator= 2023-09-22T13:17:58.122 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.027+0000 7fd98c0da500 20 chain_cache_entry: couldn't find cache locator 2023-09-22T13:17:58.122 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.027+0000 7fd98c0da500 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-09-22T13:17:58.122 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.028+0000 7fd98c0da500 20 get_obj_state: rctx=0x560fea23f370 obj=check-bucket:e state=0x560fea285258 s->prefetch_data=0 2023-09-22T13:17:58.123 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.028+0000 7fd98c0da500 20 bucket_index_guard_olh_op(): olh_state.olh_tag=i411k1e0cez4bize043aabgvjc1i0i92 2023-09-22T13:17:58.123 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.029+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:58.123 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.030+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:58.123 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.030+0000 7fd98c0da500 10 cache get: name=default.rgw.log++bucket.sync-source-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:58.123 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.030+0000 7fd98c0da500 10 cache get: name=default.rgw.log++bucket.sync-target-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:58.123 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.030+0000 7fd98c0da500 10 chain_cache_entry: cache_locator= 2023-09-22T13:17:58.123 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.030+0000 7fd98c0da500 20 chain_cache_entry: couldn't find cache locator 2023-09-22T13:17:58.123 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.030+0000 7fd98c0da500 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-09-22T13:17:58.123 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.030+0000 7fd98c0da500 1 NOTICE: finished shard 1 (1 entries removed) 2023-09-22T13:17:58.123 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.030+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.123 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.031+0000 7fd98c0da500 20 get_obj_state: rctx=0x560fea263570 obj=check-bucket:d state=0x560fea285258 s->prefetch_data=0 2023-09-22T13:17:58.123 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.031+0000 7fd98c0da500 20 get_obj_state: setting s->obj_tag to 5ae0tg9nugofps6h1wwyb1pqu0favgeh 2023-09-22T13:17:58.124 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.031+0000 7fd98c0da500 20 get_obj_state_impl: setting s->olh_tag to 7qsd3679ddkkjzkav8aiagm75tlvzsq5 2023-09-22T13:17:58.124 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.031+0000 7fd98c0da500 20 bucket_index_guard_olh_op(): olh_state.olh_tag=7qsd3679ddkkjzkav8aiagm75tlvzsq5 2023-09-22T13:17:58.124 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.032+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.124 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.033+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.124 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.033+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.124 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.034+0000 7fd98c0da500 20 olh_log_entry: epoch=3 op=3 key=d[b7MKyGBgCBBgH505pyfuBr4icD1P96C] 2023-09-22T13:17:58.124 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.034+0000 7fd98c0da500 20 get_obj_state: rctx=0x560fea263570 obj=check-bucket:_:b7MKyGBgCBBgH505pyfuBr4icD1P96C_d state=0x560fea2b43d8 s->prefetch_data=0 2023-09-22T13:17:58.124 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.034+0000 7fd98c0da500 10 manifest: total_size = 12 2023-09-22T13:17:58.124 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.034+0000 7fd98c0da500 20 get_obj_state: setting s->obj_tag to 42123edd-4a97-43ec-80bc-fc5350f88c11.4527.3543448438409866281 2023-09-22T13:17:58.124 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.034+0000 7fd98c0da500 20 get_obj_state: rctx=0x560fea263570 obj=check-bucket:_:b7MKyGBgCBBgH505pyfuBr4icD1P96C_d state=0x560fea2b43d8 s->prefetch_data=0 2023-09-22T13:17:58.124 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.034+0000 7fd98c0da500 20 prepare_atomic_modification: state is not atomic. state=0x560fea2b43d8 2023-09-22T13:17:58.124 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.034+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.125 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.036+0000 7fd98c0da500 10 cache get: name=default.rgw.log++bucket.sync-source-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:58.125 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.036+0000 7fd98c0da500 10 cache get: name=default.rgw.log++bucket.sync-target-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:58.125 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.036+0000 7fd98c0da500 10 chain_cache_entry: cache_locator= 2023-09-22T13:17:58.125 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.036+0000 7fd98c0da500 20 chain_cache_entry: couldn't find cache locator 2023-09-22T13:17:58.318 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.036+0000 7fd98c0da500 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-09-22T13:17:58.318 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.037+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.318 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.037+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.318 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.038+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.318 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.038+0000 7fd98c0da500 10 cache get: name=default.rgw.log++bucket.sync-source-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:58.318 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.038+0000 7fd98c0da500 10 cache get: name=default.rgw.log++bucket.sync-target-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:58.318 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.038+0000 7fd98c0da500 10 chain_cache_entry: cache_locator= 2023-09-22T13:17:58.318 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.038+0000 7fd98c0da500 20 chain_cache_entry: couldn't find cache locator 2023-09-22T13:17:58.319 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.038+0000 7fd98c0da500 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-09-22T13:17:58.319 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.038+0000 7fd98c0da500 1 NOTICE: finished shard 5 (1 entries removed) 2023-09-22T13:17:58.319 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.038+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.319 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.038+0000 7fd98c0da500 20 get_obj_state: rctx=0x560fea284070 obj=check-bucket:c state=0x560fea285258 s->prefetch_data=0 2023-09-22T13:17:58.319 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.038+0000 7fd98c0da500 20 get_obj_state: setting s->obj_tag to jogkk1lyeocreujbycjtrtll8bgivcxm 2023-09-22T13:17:58.319 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.038+0000 7fd98c0da500 20 get_obj_state_impl: setting s->olh_tag to pww24tiki3p98t3vt06ip99sghububrl 2023-09-22T13:17:58.319 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.038+0000 7fd98c0da500 20 bucket_index_guard_olh_op(): olh_state.olh_tag=pww24tiki3p98t3vt06ip99sghububrl 2023-09-22T13:17:58.319 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.039+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.319 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.040+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.319 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.040+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.319 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.041+0000 7fd98c0da500 20 olh_log_entry: epoch=3 op=3 key=c[yb9Hr2BGp0G-kxxflimnR4LitNGAN41] 2023-09-22T13:17:58.319 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.041+0000 7fd98c0da500 20 get_obj_state: rctx=0x560fea284070 obj=check-bucket:_:yb9Hr2BGp0G-kxxflimnR4LitNGAN41_c state=0x560fea2ad2f8 s->prefetch_data=0 2023-09-22T13:17:58.320 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.041+0000 7fd98c0da500 10 manifest: total_size = 9 2023-09-22T13:17:58.320 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.041+0000 7fd98c0da500 20 get_obj_state: setting s->obj_tag to 42123edd-4a97-43ec-80bc-fc5350f88c11.4527.3951062308233096839 2023-09-22T13:17:58.320 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.041+0000 7fd98c0da500 20 get_obj_state: rctx=0x560fea284070 obj=check-bucket:_:yb9Hr2BGp0G-kxxflimnR4LitNGAN41_c state=0x560fea2ad2f8 s->prefetch_data=0 2023-09-22T13:17:58.320 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.041+0000 7fd98c0da500 20 prepare_atomic_modification: state is not atomic. state=0x560fea2ad2f8 2023-09-22T13:17:58.320 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.041+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.320 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.043+0000 7fd98c0da500 10 cache get: name=default.rgw.log++bucket.sync-source-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:58.320 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.043+0000 7fd98c0da500 10 cache get: name=default.rgw.log++bucket.sync-target-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:58.320 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.043+0000 7fd98c0da500 10 chain_cache_entry: cache_locator= 2023-09-22T13:17:58.320 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.043+0000 7fd98c0da500 20 chain_cache_entry: couldn't find cache locator 2023-09-22T13:17:58.320 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.043+0000 7fd98c0da500 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-09-22T13:17:58.320 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.044+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.320 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.044+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.321 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.045+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.321 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.045+0000 7fd98c0da500 10 cache get: name=default.rgw.log++bucket.sync-source-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:58.321 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.045+0000 7fd98c0da500 10 cache get: name=default.rgw.log++bucket.sync-target-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:58.321 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.045+0000 7fd98c0da500 10 chain_cache_entry: cache_locator= 2023-09-22T13:17:58.321 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.045+0000 7fd98c0da500 20 chain_cache_entry: couldn't find cache locator 2023-09-22T13:17:58.321 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.045+0000 7fd98c0da500 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-09-22T13:17:58.321 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.045+0000 7fd98c0da500 1 NOTICE: finished shard 8 (1 entries removed) 2023-09-22T13:17:58.321 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.045+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:58.321 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.045+0000 7fd98c0da500 20 get_obj_state: rctx=0x560fea2a5270 obj=check-bucket:f state=0x560fea285258 s->prefetch_data=0 2023-09-22T13:17:58.321 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.046+0000 7fd98c0da500 20 get_obj_state: setting s->obj_tag to 4ye32kvajraojq5pf3p92pgo6xvl76v2 2023-09-22T13:17:58.321 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.046+0000 7fd98c0da500 20 get_obj_state_impl: setting s->olh_tag to t24qx6v72k2g9uxqboq5wva1a5vuicbz 2023-09-22T13:17:58.321 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.046+0000 7fd98c0da500 20 bucket_index_guard_olh_op(): olh_state.olh_tag=t24qx6v72k2g9uxqboq5wva1a5vuicbz 2023-09-22T13:17:58.321 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.046+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:58.322 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.047+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:58.322 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.047+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:58.322 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.048+0000 7fd98c0da500 20 olh_log_entry: epoch=2 op=2 key=f[] 2023-09-22T13:17:58.322 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.048+0000 7fd98c0da500 20 olh_log_entry: epoch=2 op=3 key=f[OLiWUE0tYmzbHz775hS7Uguungxut2W] 2023-09-22T13:17:58.322 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.048+0000 7fd98c0da500 20 get_obj_state: rctx=0x560fea2a5270 obj=check-bucket:_:OLiWUE0tYmzbHz775hS7Uguungxut2W_f state=0x560fea2ad2f8 s->prefetch_data=0 2023-09-22T13:17:58.322 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.048+0000 7fd98c0da500 10 manifest: total_size = 22 2023-09-22T13:17:58.322 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.048+0000 7fd98c0da500 20 get_obj_state: setting s->obj_tag to 42123edd-4a97-43ec-80bc-fc5350f88c11.4527.1187727183162884347 2023-09-22T13:17:58.322 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.048+0000 7fd98c0da500 20 get_obj_state: rctx=0x560fea2a5270 obj=check-bucket:_:OLiWUE0tYmzbHz775hS7Uguungxut2W_f state=0x560fea2ad2f8 s->prefetch_data=0 2023-09-22T13:17:58.322 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.048+0000 7fd98c0da500 20 prepare_atomic_modification: state is not atomic. state=0x560fea2ad2f8 2023-09-22T13:17:58.322 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.048+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:58.322 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.050+0000 7fd98c0da500 10 cache get: name=default.rgw.log++bucket.sync-source-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:58.323 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.050+0000 7fd98c0da500 10 cache get: name=default.rgw.log++bucket.sync-target-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:58.323 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.050+0000 7fd98c0da500 10 chain_cache_entry: cache_locator= 2023-09-22T13:17:58.323 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.050+0000 7fd98c0da500 20 chain_cache_entry: couldn't find cache locator 2023-09-22T13:17:58.323 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.050+0000 7fd98c0da500 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-09-22T13:17:58.323 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.051+0000 7fd98c0da500 20 get_obj_state: rctx=0x560fea2a5270 obj=check-bucket:f state=0x560fea285258 s->prefetch_data=0 2023-09-22T13:17:58.323 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.051+0000 7fd98c0da500 20 bucket_index_guard_olh_op(): olh_state.olh_tag=t24qx6v72k2g9uxqboq5wva1a5vuicbz 2023-09-22T13:17:58.323 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.053+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:58.323 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.054+0000 7fd98c0da500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:58.323 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.054+0000 7fd98c0da500 10 cache get: name=default.rgw.log++bucket.sync-source-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:58.323 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.054+0000 7fd98c0da500 10 cache get: name=default.rgw.log++bucket.sync-target-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:58.323 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.054+0000 7fd98c0da500 10 chain_cache_entry: cache_locator= 2023-09-22T13:17:58.324 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.054+0000 7fd98c0da500 20 chain_cache_entry: couldn't find cache locator 2023-09-22T13:17:58.324 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.054+0000 7fd98c0da500 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-09-22T13:17:58.324 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.054+0000 7fd98c0da500 1 NOTICE: finished shard 10 (1 entries removed) 2023-09-22T13:17:58.324 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.054+0000 7fd98c0da500 1 NOTICE: finished all shards (4 entries removed) 2023-09-22T13:17:58.324 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.055+0000 7fd98c0da500 20 remove_watcher() i=0 2023-09-22T13:17:58.324 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.055+0000 7fd98c0da500 2 removed watcher, disabling cache 2023-09-22T13:17:58.324 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.056+0000 7fd98c0da500 20 remove_watcher() i=1 2023-09-22T13:17:58.324 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.057+0000 7fd98c0da500 20 remove_watcher() i=2 2023-09-22T13:17:58.324 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.057+0000 7fd98c0da500 20 remove_watcher() i=3 2023-09-22T13:17:58.324 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.058+0000 7fd98c0da500 20 remove_watcher() i=4 2023-09-22T13:17:58.324 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.059+0000 7fd98c0da500 20 remove_watcher() i=5 2023-09-22T13:17:58.324 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.060+0000 7fd98c0da500 20 remove_watcher() i=6 2023-09-22T13:17:58.325 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.060+0000 7fd98c0da500 20 remove_watcher() i=7 2023-09-22T13:17:58.325 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.061+0000 7fd8c3fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.325 INFO:tasks.workunit.client.0.smithi096.stderr:TEST: bucket check unlinked does not affect normal entries 2023-09-22T13:17:58.325 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:58.325 INFO:tasks.workunit.client.0.smithi096.stderr:TEST: bucket check unlinked does not find new unlistable entries after fix 2023-09-22T13:17:58.325 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:58.325 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --min-age-hours 0 --dump-keys 2023-09-22T13:17:58.325 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:58.325 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:58.325 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.169+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.325 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.171+0000 7f34c7e6f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.325 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.171+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.326 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.172+0000 7f34c7e6f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.326 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.172+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.326 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.172+0000 7f34c7e6f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.326 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.172+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.326 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.173+0000 7f34c7e6f500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.326 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.182+0000 7f34c7e6f500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:58.326 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.182+0000 7f34c7e6f500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:58.326 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.182+0000 7f34c7e6f500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:58.326 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.182+0000 7f34c7e6f500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:58.326 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.182+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.326 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.183+0000 7f34c7e6f500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.327 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.183+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.327 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.183+0000 7f34c7e6f500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.327 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.183+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.327 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.183+0000 7f34c7e6f500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.327 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.183+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.327 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.183+0000 7f34c7e6f500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.327 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.183+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.327 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.183+0000 7f34c7e6f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.327 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.183+0000 7f34c7e6f500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:58.327 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.183+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.327 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.184+0000 7f34c7e6f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.327 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.184+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.327 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.184+0000 7f34c7e6f500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.327 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.184+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.328 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.184+0000 7f34c7e6f500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.328 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.184+0000 7f34c7e6f500 20 zonegroup default 2023-09-22T13:17:58.328 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.184+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.328 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.185+0000 7f34c7e6f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.328 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.185+0000 7f34c7e6f500 10 Cannot find current period zone using local zone 2023-09-22T13:17:58.328 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.185+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.328 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.185+0000 7f34c7e6f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.328 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.185+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.328 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.185+0000 7f34c7e6f500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.328 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.185+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.328 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.185+0000 7f34c7e6f500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.328 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.185+0000 7f34c7e6f500 20 zone default found 2023-09-22T13:17:58.328 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.185+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.329 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.186+0000 7f34c7e6f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.329 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.186+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.329 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.186+0000 7f34c7e6f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.329 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.186+0000 7f34c7e6f500 20 started sync module instance, tier type = 2023-09-22T13:17:58.329 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.186+0000 7f34c7e6f500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:58.329 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.192+0000 7f34c7e6f500 20 add_watcher() i=0 2023-09-22T13:17:58.329 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.192+0000 7f34c7e6f500 20 add_watcher() i=1 2023-09-22T13:17:58.329 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.192+0000 7f34c7e6f500 20 add_watcher() i=2 2023-09-22T13:17:58.329 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.192+0000 7f34c7e6f500 20 add_watcher() i=3 2023-09-22T13:17:58.329 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.192+0000 7f34c7e6f500 20 add_watcher() i=4 2023-09-22T13:17:58.329 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.192+0000 7f34c7e6f500 20 add_watcher() i=5 2023-09-22T13:17:58.329 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.192+0000 7f34c7e6f500 20 add_watcher() i=6 2023-09-22T13:17:58.329 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.193+0000 7f34b3351700 0 -- 172.21.15.96:0/2748065674 >> [v2:172.21.15.96:6807/140797,v1:172.21.15.96:6809/140797] conn(0x7f34a401c650 msgr2=0x7f34a401ca70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:58.330 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.193+0000 7f34c7e6f500 20 add_watcher() i=7 2023-09-22T13:17:58.330 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.193+0000 7f34c7e6f500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:58.330 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.194+0000 7f33fffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.330 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.194+0000 7f34c7e6f500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:58.330 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.194+0000 7f34c7e6f500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:58.330 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.198+0000 7f34b2b50700 0 -- 172.21.15.96:0/3258327161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5598447ca940 msgr2=0x5598447cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-09-22T13:17:58.330 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.200+0000 7f34c7e6f500 5 note: GC not initialized 2023-09-22T13:17:58.330 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.201+0000 7f33eeffd700 20 reqs_thread_entry: start 2023-09-22T13:17:58.330 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.201+0000 7f33ed7fa700 20 reqs_thread_entry: start 2023-09-22T13:17:58.330 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.201+0000 7f34c7e6f500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:58.330 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.201+0000 7f34c7e6f500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:58.330 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.201+0000 7f33e77fe700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:23 2023 2023-09-22T13:17:58.330 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:58.331 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.201+0000 7f33e6ffd700 20 reqs_thread_entry: start 2023-09-22T13:17:58.331 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.201+0000 7f34c7e6f500 20 get_system_obj_state: rctx=0x7ffc5af1e378 obj=default.rgw.meta:root:check-bucket state=0x5598447ca4d0 s->prefetch_data=0 2023-09-22T13:17:58.331 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.201+0000 7f34c7e6f500 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-09-22T13:17:58.331 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.202+0000 7f34c7e6f500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x16 2023-09-22T13:17:58.331 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.202+0000 7f34c7e6f500 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:58.331 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.202+0000 7f34c7e6f500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:58.331 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.202+0000 7f34c7e6f500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.331 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.202+0000 7f34c7e6f500 10 cache get: name=default.rgw.meta+root+check-bucket : type miss (requested=0x11, cached=0x16) 2023-09-22T13:17:58.331 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.202+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.331 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.202+0000 7f34c7e6f500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:58.331 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.202+0000 7f34c7e6f500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-09-22T13:17:58.331 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.202+0000 7f34c7e6f500 10 moving default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:58.332 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.202+0000 7f34c7e6f500 20 get_system_obj_state: rctx=0x7ffc5af1e448 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5598447ca4d0 s->prefetch_data=0 2023-09-22T13:17:58.332 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.202+0000 7f34c7e6f500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : miss 2023-09-22T13:17:58.332 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x16 2023-09-22T13:17:58.332 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:58.332 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:58.332 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:58.332 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.332 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.332 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.332 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : type miss (requested=0x13, cached=0x16) 2023-09-22T13:17:58.332 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.332 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.332 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x13 2023-09-22T13:17:58.333 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 10 moving default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:58.333 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:58.333 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:58.333 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 2023-09-22T13:17:58.333 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:58.333 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 1 NOTICE: finished shard 0 (0 entries found) 2023-09-22T13:17:58.333 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.203+0000 7f34c7e6f500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:58.333 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.204+0000 7f34c7e6f500 1 NOTICE: finished shard 1 (0 entries found) 2023-09-22T13:17:58.333 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.204+0000 7f34c7e6f500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:58.333 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.205+0000 7f34c7e6f500 1 NOTICE: finished shard 2 (0 entries found) 2023-09-22T13:17:58.333 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.205+0000 7f34c7e6f500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:58.334 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.205+0000 7f34c7e6f500 1 NOTICE: finished shard 3 (0 entries found) 2023-09-22T13:17:58.334 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.205+0000 7f34c7e6f500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:58.334 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.206+0000 7f34c7e6f500 1 NOTICE: finished shard 4 (0 entries found) 2023-09-22T13:17:58.334 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.206+0000 7f34c7e6f500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.334 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.206+0000 7f34c7e6f500 1 NOTICE: finished shard 5 (0 entries found) 2023-09-22T13:17:58.334 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.206+0000 7f34c7e6f500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:58.334 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.207+0000 7f34c7e6f500 1 NOTICE: finished shard 6 (0 entries found) 2023-09-22T13:17:58.334 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.207+0000 7f34c7e6f500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:58.334 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.207+0000 7f34c7e6f500 1 NOTICE: finished shard 7 (0 entries found) 2023-09-22T13:17:58.334 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.207+0000 7f34c7e6f500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.335 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.208+0000 7f34c7e6f500 1 NOTICE: finished shard 8 (0 entries found) 2023-09-22T13:17:58.335 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.208+0000 7f34c7e6f500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:58.335 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.208+0000 7f34c7e6f500 1 NOTICE: finished shard 9 (0 entries found) 2023-09-22T13:17:58.335 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.208+0000 7f34c7e6f500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:58.335 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.209+0000 7f34c7e6f500 1 NOTICE: finished shard 10 (0 entries found) 2023-09-22T13:17:58.335 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.209+0000 7f34c7e6f500 1 NOTICE: finished all shards (0 entries found) 2023-09-22T13:17:58.335 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.210+0000 7f34c7e6f500 20 remove_watcher() i=0 2023-09-22T13:17:58.335 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.210+0000 7f34c7e6f500 2 removed watcher, disabling cache 2023-09-22T13:17:58.335 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.211+0000 7f34c7e6f500 20 remove_watcher() i=1 2023-09-22T13:17:58.335 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.211+0000 7f34c7e6f500 20 remove_watcher() i=2 2023-09-22T13:17:58.336 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.212+0000 7f34c7e6f500 20 remove_watcher() i=3 2023-09-22T13:17:58.336 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.213+0000 7f34c7e6f500 20 remove_watcher() i=4 2023-09-22T13:17:58.336 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.214+0000 7f34c7e6f500 20 remove_watcher() i=5 2023-09-22T13:17:58.336 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.214+0000 7f34c7e6f500 20 remove_watcher() i=6 2023-09-22T13:17:58.336 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.215+0000 7f34c7e6f500 20 remove_watcher() i=7 2023-09-22T13:17:58.336 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.216+0000 7f33fffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.336 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bi list --bucket check-bucket --object c 2023-09-22T13:17:58.336 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:58.336 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:58.336 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.276+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.337 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.278+0000 7f0d78650500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.337 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.278+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.337 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.279+0000 7f0d78650500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.337 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.279+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.337 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.279+0000 7f0d78650500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.337 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.279+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.337 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.280+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.337 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.289+0000 7f0d78650500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:58.337 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.289+0000 7f0d78650500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:58.337 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.289+0000 7f0d78650500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:58.337 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.289+0000 7f0d78650500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:58.338 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.289+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.338 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.289+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.338 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.289+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.338 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.289+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.338 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.289+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.338 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.290+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.338 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.290+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.338 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.290+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.338 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.290+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.338 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.290+0000 7f0d78650500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.338 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.290+0000 7f0d78650500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:58.338 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.290+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.290+0000 7f0d78650500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.290+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.291+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.291+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.291+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.291+0000 7f0d78650500 20 zonegroup default 2023-09-22T13:17:58.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.291+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.291+0000 7f0d78650500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.291+0000 7f0d78650500 10 Cannot find current period zone using local zone 2023-09-22T13:17:58.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.291+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.291+0000 7f0d78650500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.291+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.291+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.291+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.292+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.292+0000 7f0d78650500 20 zone default found 2023-09-22T13:17:58.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.292+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.292+0000 7f0d78650500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.292+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.292+0000 7f0d78650500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.292+0000 7f0d78650500 20 started sync module instance, tier type = 2023-09-22T13:17:58.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.292+0000 7f0d78650500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:58.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.299+0000 7f0d78650500 20 add_watcher() i=0 2023-09-22T13:17:58.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.299+0000 7f0d78650500 20 add_watcher() i=1 2023-09-22T13:17:58.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.299+0000 7f0d78650500 20 add_watcher() i=2 2023-09-22T13:17:58.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.299+0000 7f0d78650500 20 add_watcher() i=3 2023-09-22T13:17:58.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.299+0000 7f0d78650500 20 add_watcher() i=4 2023-09-22T13:17:58.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.299+0000 7f0d78650500 20 add_watcher() i=5 2023-09-22T13:17:58.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.299+0000 7f0d78650500 20 add_watcher() i=6 2023-09-22T13:17:58.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.300+0000 7f0d63b32700 0 -- 172.21.15.96:0/985486849 >> [v2:172.21.15.96:6807/140797,v1:172.21.15.96:6809/140797] conn(0x7f0d4c01e440 msgr2=0x7f0d4c01e880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:58.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.300+0000 7f0d78650500 20 add_watcher() i=7 2023-09-22T13:17:58.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.300+0000 7f0d78650500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:58.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.302+0000 7f0d357e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.302+0000 7f0d78650500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:58.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.302+0000 7f0d78650500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:58.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.306+0000 7f0d63b32700 0 -- 172.21.15.96:0/2012613255 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55a41d44cdf0 msgr2=0x55a41d420520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:58.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.308+0000 7f0d78650500 5 note: GC not initialized 2023-09-22T13:17:58.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.309+0000 7f0ca3fff700 20 reqs_thread_entry: start 2023-09-22T13:17:58.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.309+0000 7f0ca1ffb700 20 reqs_thread_entry: start 2023-09-22T13:17:58.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.309+0000 7f0d78650500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:58.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.309+0000 7f0d78650500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:58.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.309+0000 7f0c8ffff700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:23 2023 2023-09-22T13:17:58.342 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:58.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.309+0000 7f0c8f7fe700 20 reqs_thread_entry: start 2023-09-22T13:17:58.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.309+0000 7f0d78650500 20 get_system_obj_state: rctx=0x7ffd2aebe150 obj=default.rgw.meta:root:check-bucket state=0x55a41d44c710 s->prefetch_data=0 2023-09-22T13:17:58.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.309+0000 7f0d78650500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.309+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.310+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:58.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.310+0000 7f0d78650500 20 get_system_obj_state: rctx=0x7ffd2aebe150 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55a41d41e610 s->prefetch_data=0 2023-09-22T13:17:58.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.310+0000 7f0d78650500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.310+0000 7f0d78650500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.310+0000 7f0d78650500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.310+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.310+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.310+0000 7f0d78650500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.311+0000 7f0d78650500 20 get_system_obj_state: rctx=0x7ffd2aebdf20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55a41d41e610 s->prefetch_data=0 2023-09-22T13:17:58.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.311+0000 7f0d78650500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.311+0000 7f0d78650500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.311+0000 7f0d78650500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.311+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.311+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.344 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.311+0000 7f0d78650500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.344 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.311+0000 7f0d78650500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:58.344 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.312+0000 7f0d78650500 20 get_system_obj_state: rctx=0x7ffd2aebdf20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55a41d41e610 s->prefetch_data=0 2023-09-22T13:17:58.344 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.312+0000 7f0d78650500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.312+0000 7f0d78650500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.312+0000 7f0d78650500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.312+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.312+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.312+0000 7f0d78650500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.312+0000 7f0d78650500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:58.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.313+0000 7f0d78650500 20 get_system_obj_state: rctx=0x7ffd2aebdf20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55a41d41e610 s->prefetch_data=0 2023-09-22T13:17:58.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.314+0000 7f0d78650500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.314+0000 7f0d78650500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.314+0000 7f0d78650500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.314+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.314+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.314+0000 7f0d78650500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.314+0000 7f0d78650500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:58.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.315+0000 7f0d78650500 20 get_system_obj_state: rctx=0x7ffd2aebdf20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55a41d41e610 s->prefetch_data=0 2023-09-22T13:17:58.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.315+0000 7f0d78650500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.315+0000 7f0d78650500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.315+0000 7f0d78650500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.315+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.316+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.316+0000 7f0d78650500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.316+0000 7f0d78650500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:58.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.316+0000 7f0d78650500 20 get_system_obj_state: rctx=0x7ffd2aebdf20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55a41d41e610 s->prefetch_data=0 2023-09-22T13:17:58.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.316+0000 7f0d78650500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.316+0000 7f0d78650500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.316+0000 7f0d78650500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.316+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.317+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.317+0000 7f0d78650500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.317+0000 7f0d78650500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:58.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.317+0000 7f0d78650500 20 get_system_obj_state: rctx=0x7ffd2aebdf20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55a41d41e610 s->prefetch_data=0 2023-09-22T13:17:58.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.318+0000 7f0d78650500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.318+0000 7f0d78650500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.318+0000 7f0d78650500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.318+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.318+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.318+0000 7f0d78650500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.318+0000 7f0d78650500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.319+0000 7f0d78650500 20 get_system_obj_state: rctx=0x7ffd2aebdf20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55a41d41e610 s->prefetch_data=0 2023-09-22T13:17:58.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.319+0000 7f0d78650500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.319+0000 7f0d78650500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.319+0000 7f0d78650500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.319+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.319+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.319+0000 7f0d78650500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.319+0000 7f0d78650500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:58.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.320+0000 7f0d78650500 20 get_system_obj_state: rctx=0x7ffd2aebdf20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55a41d41e610 s->prefetch_data=0 2023-09-22T13:17:58.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.321+0000 7f0d78650500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.321+0000 7f0d78650500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.321+0000 7f0d78650500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.321+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.321+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.321+0000 7f0d78650500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.321+0000 7f0d78650500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:58.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.321+0000 7f0d78650500 20 get_system_obj_state: rctx=0x7ffd2aebdf20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55a41d41e610 s->prefetch_data=0 2023-09-22T13:17:58.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.322+0000 7f0d78650500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.322+0000 7f0d78650500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.322+0000 7f0d78650500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.322+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.322+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.322+0000 7f0d78650500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.322+0000 7f0d78650500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.539 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.323+0000 7f0d78650500 20 get_system_obj_state: rctx=0x7ffd2aebdf20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55a41d41e610 s->prefetch_data=0 2023-09-22T13:17:58.539 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.323+0000 7f0d78650500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.539 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.323+0000 7f0d78650500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.539 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.323+0000 7f0d78650500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.539 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.323+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.539 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.323+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.539 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.323+0000 7f0d78650500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.539 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.323+0000 7f0d78650500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:58.539 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.324+0000 7f0d78650500 20 get_system_obj_state: rctx=0x7ffd2aebdf20 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55a41d41e610 s->prefetch_data=0 2023-09-22T13:17:58.539 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.324+0000 7f0d78650500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.539 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.324+0000 7f0d78650500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.539 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.324+0000 7f0d78650500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.540 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.324+0000 7f0d78650500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.540 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.325+0000 7f0d78650500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.540 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.325+0000 7f0d78650500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.540 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.325+0000 7f0d78650500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:58.540 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.326+0000 7f0d78650500 20 remove_watcher() i=0 2023-09-22T13:17:58.540 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.326+0000 7f0d78650500 2 removed watcher, disabling cache 2023-09-22T13:17:58.540 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.327+0000 7f0d78650500 20 remove_watcher() i=1 2023-09-22T13:17:58.540 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.328+0000 7f0d78650500 20 remove_watcher() i=2 2023-09-22T13:17:58.540 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.328+0000 7f0d78650500 20 remove_watcher() i=3 2023-09-22T13:17:58.540 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.329+0000 7f0d78650500 20 remove_watcher() i=4 2023-09-22T13:17:58.540 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.330+0000 7f0d78650500 20 remove_watcher() i=5 2023-09-22T13:17:58.540 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.331+0000 7f0d78650500 20 remove_watcher() i=6 2023-09-22T13:17:58.541 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.331+0000 7f0d78650500 20 remove_watcher() i=7 2023-09-22T13:17:58.541 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.332+0000 7f0d357e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.541 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bi list --bucket check-bucket --object d 2023-09-22T13:17:58.541 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:58.541 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:58.541 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.370+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.541 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.371+0000 7fa9c0e18500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.541 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.371+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.541 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.372+0000 7fa9c0e18500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.541 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.372+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.541 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.372+0000 7fa9c0e18500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.541 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.372+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.542 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.373+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.542 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.382+0000 7fa9c0e18500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:58.542 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.382+0000 7fa9c0e18500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:58.542 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.382+0000 7fa9c0e18500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:58.542 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.382+0000 7fa9c0e18500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:58.542 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.382+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.542 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.383+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.542 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.383+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.542 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.383+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.542 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.383+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.542 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.383+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.543 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.383+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.543 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.383+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.543 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.383+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.543 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.384+0000 7fa9c0e18500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.543 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.384+0000 7fa9c0e18500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:58.543 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.384+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.543 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.384+0000 7fa9c0e18500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.543 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.384+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.543 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.384+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.543 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.384+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.543 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.384+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.543 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.384+0000 7fa9c0e18500 20 zonegroup default 2023-09-22T13:17:58.544 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.384+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.544 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.384+0000 7fa9c0e18500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.544 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.384+0000 7fa9c0e18500 10 Cannot find current period zone using local zone 2023-09-22T13:17:58.544 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.384+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.544 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.385+0000 7fa9c0e18500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.544 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.385+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.544 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.385+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.544 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.385+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.544 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.385+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.544 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.385+0000 7fa9c0e18500 20 zone default found 2023-09-22T13:17:58.544 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.385+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.544 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.385+0000 7fa9c0e18500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.545 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.385+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.545 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.386+0000 7fa9c0e18500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.545 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.386+0000 7fa9c0e18500 20 started sync module instance, tier type = 2023-09-22T13:17:58.545 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.386+0000 7fa9c0e18500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:58.545 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.392+0000 7fa9ac2fa700 0 -- 172.21.15.96:0/2157906266 >> [v2:172.21.15.96:6807/140797,v1:172.21.15.96:6809/140797] conn(0x7fa99c04d7a0 msgr2=0x7fa99c04c340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:58.545 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.392+0000 7fa9c0e18500 20 add_watcher() i=0 2023-09-22T13:17:58.545 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.392+0000 7fa9c0e18500 20 add_watcher() i=1 2023-09-22T13:17:58.545 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.392+0000 7fa9c0e18500 20 add_watcher() i=2 2023-09-22T13:17:58.545 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.392+0000 7fa9c0e18500 20 add_watcher() i=3 2023-09-22T13:17:58.545 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.392+0000 7fa9c0e18500 20 add_watcher() i=4 2023-09-22T13:17:58.545 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.392+0000 7fa9c0e18500 20 add_watcher() i=5 2023-09-22T13:17:58.545 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.392+0000 7fa9c0e18500 20 add_watcher() i=6 2023-09-22T13:17:58.546 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.393+0000 7fa9c0e18500 20 add_watcher() i=7 2023-09-22T13:17:58.546 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.393+0000 7fa9c0e18500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:58.546 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.394+0000 7fa97dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.546 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.395+0000 7fa9c0e18500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:58.546 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.395+0000 7fa9c0e18500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:58.546 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.398+0000 7fa9abaf9700 0 -- 172.21.15.96:0/1647327503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x561c315cc3a0 msgr2=0x561c315d0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:58.546 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.400+0000 7fa9c0e18500 5 note: GC not initialized 2023-09-22T13:17:58.546 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.401+0000 7fa8f4ff9700 20 reqs_thread_entry: start 2023-09-22T13:17:58.546 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.401+0000 7fa8ea7fc700 20 reqs_thread_entry: start 2023-09-22T13:17:58.546 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.401+0000 7fa9c0e18500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:58.546 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.402+0000 7fa9c0e18500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:58.546 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.402+0000 7fa8e8ff9700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:23 2023 2023-09-22T13:17:58.547 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:58.547 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.402+0000 7fa8cffff700 20 reqs_thread_entry: start 2023-09-22T13:17:58.547 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.402+0000 7fa9c0e18500 20 get_system_obj_state: rctx=0x7ffc56d11c90 obj=default.rgw.meta:root:check-bucket state=0x561c315b56a0 s->prefetch_data=0 2023-09-22T13:17:58.547 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.402+0000 7fa9c0e18500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.547 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.402+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.547 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.402+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:58.547 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.402+0000 7fa9c0e18500 20 get_system_obj_state: rctx=0x7ffc56d11c90 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x561c3152d610 s->prefetch_data=0 2023-09-22T13:17:58.547 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.403+0000 7fa9c0e18500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.547 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.403+0000 7fa9c0e18500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.547 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.403+0000 7fa9c0e18500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.547 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.403+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.547 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.403+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.547 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.403+0000 7fa9c0e18500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.548 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.403+0000 7fa9c0e18500 20 get_system_obj_state: rctx=0x7ffc56d11a60 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x561c3152d610 s->prefetch_data=0 2023-09-22T13:17:58.548 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.403+0000 7fa9c0e18500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.548 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.403+0000 7fa9c0e18500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.548 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.403+0000 7fa9c0e18500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.548 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.403+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.548 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.404+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.548 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.404+0000 7fa9c0e18500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.548 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.404+0000 7fa9c0e18500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:58.548 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.404+0000 7fa9c0e18500 20 get_system_obj_state: rctx=0x7ffc56d11a60 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x561c3152d610 s->prefetch_data=0 2023-09-22T13:17:58.548 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.404+0000 7fa9c0e18500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.548 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.404+0000 7fa9c0e18500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.548 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.404+0000 7fa9c0e18500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.549 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.404+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.549 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.405+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.549 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.405+0000 7fa9c0e18500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.549 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.405+0000 7fa9c0e18500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:58.549 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.405+0000 7fa9c0e18500 20 get_system_obj_state: rctx=0x7ffc56d11a60 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x561c3152d610 s->prefetch_data=0 2023-09-22T13:17:58.549 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.406+0000 7fa9c0e18500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.549 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.406+0000 7fa9c0e18500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.549 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.406+0000 7fa9c0e18500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.549 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.406+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.549 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.406+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.549 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.406+0000 7fa9c0e18500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.549 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.406+0000 7fa9c0e18500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:58.550 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.406+0000 7fa9c0e18500 20 get_system_obj_state: rctx=0x7ffc56d11a60 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x561c3152d610 s->prefetch_data=0 2023-09-22T13:17:58.550 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.407+0000 7fa9c0e18500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.550 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.407+0000 7fa9c0e18500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.550 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.407+0000 7fa9c0e18500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.550 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.407+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.550 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.407+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.550 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.407+0000 7fa9c0e18500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.550 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.407+0000 7fa9c0e18500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:58.550 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.408+0000 7fa9c0e18500 20 get_system_obj_state: rctx=0x7ffc56d11a60 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x561c3152d610 s->prefetch_data=0 2023-09-22T13:17:58.550 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.408+0000 7fa9c0e18500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.550 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.408+0000 7fa9c0e18500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.550 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.408+0000 7fa9c0e18500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.551 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.408+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.551 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.408+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.551 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.408+0000 7fa9c0e18500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.551 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.408+0000 7fa9c0e18500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:58.551 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.409+0000 7fa9c0e18500 20 get_system_obj_state: rctx=0x7ffc56d11a60 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x561c3152d610 s->prefetch_data=0 2023-09-22T13:17:58.551 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.409+0000 7fa9c0e18500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.551 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.409+0000 7fa9c0e18500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.551 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.409+0000 7fa9c0e18500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.551 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.409+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.551 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.409+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.551 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.409+0000 7fa9c0e18500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.552 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.409+0000 7fa9c0e18500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.552 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.410+0000 7fa9c0e18500 20 get_system_obj_state: rctx=0x7ffc56d11a60 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x561c3152d610 s->prefetch_data=0 2023-09-22T13:17:58.552 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.410+0000 7fa9c0e18500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.552 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.410+0000 7fa9c0e18500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.552 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.410+0000 7fa9c0e18500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.552 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.410+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.552 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.411+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.552 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.411+0000 7fa9c0e18500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.552 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.411+0000 7fa9c0e18500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:58.552 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.411+0000 7fa9c0e18500 20 get_system_obj_state: rctx=0x7ffc56d11a60 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x561c3152d610 s->prefetch_data=0 2023-09-22T13:17:58.552 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.412+0000 7fa9c0e18500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.552 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.412+0000 7fa9c0e18500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.552 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.412+0000 7fa9c0e18500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.553 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.412+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.553 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.412+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.553 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.412+0000 7fa9c0e18500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.553 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.412+0000 7fa9c0e18500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:58.553 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.412+0000 7fa9c0e18500 20 get_system_obj_state: rctx=0x7ffc56d11a60 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x561c3152d610 s->prefetch_data=0 2023-09-22T13:17:58.553 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.413+0000 7fa9c0e18500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.553 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.413+0000 7fa9c0e18500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.553 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.413+0000 7fa9c0e18500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.553 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.413+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.553 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.413+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.553 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.413+0000 7fa9c0e18500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.553 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.413+0000 7fa9c0e18500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.554 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.413+0000 7fa9c0e18500 20 get_system_obj_state: rctx=0x7ffc56d11a60 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x561c3152d610 s->prefetch_data=0 2023-09-22T13:17:58.554 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.414+0000 7fa9c0e18500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.554 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.414+0000 7fa9c0e18500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.554 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.414+0000 7fa9c0e18500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.554 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.414+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.554 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.414+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.554 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.414+0000 7fa9c0e18500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.554 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.414+0000 7fa9c0e18500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:58.554 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.414+0000 7fa9c0e18500 20 get_system_obj_state: rctx=0x7ffc56d11a60 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x561c3152d610 s->prefetch_data=0 2023-09-22T13:17:58.554 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.415+0000 7fa9c0e18500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.554 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.415+0000 7fa9c0e18500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.555 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.415+0000 7fa9c0e18500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.555 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.415+0000 7fa9c0e18500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.555 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.415+0000 7fa9c0e18500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.555 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.415+0000 7fa9c0e18500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.555 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.415+0000 7fa9c0e18500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:58.555 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.416+0000 7fa9c0e18500 20 remove_watcher() i=0 2023-09-22T13:17:58.555 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.416+0000 7fa9c0e18500 2 removed watcher, disabling cache 2023-09-22T13:17:58.555 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.417+0000 7fa9c0e18500 20 remove_watcher() i=1 2023-09-22T13:17:58.555 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.418+0000 7fa9c0e18500 20 remove_watcher() i=2 2023-09-22T13:17:58.555 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.419+0000 7fa9c0e18500 20 remove_watcher() i=3 2023-09-22T13:17:58.555 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.420+0000 7fa9c0e18500 20 remove_watcher() i=4 2023-09-22T13:17:58.555 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.420+0000 7fa9c0e18500 20 remove_watcher() i=5 2023-09-22T13:17:58.555 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.421+0000 7fa9c0e18500 20 remove_watcher() i=6 2023-09-22T13:17:58.556 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.422+0000 7fa9c0e18500 20 remove_watcher() i=7 2023-09-22T13:17:58.556 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.423+0000 7fa97dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.556 INFO:tasks.workunit.client.0.smithi096.stderr:TEST: bucket check olh finds leftover OLH entries 2023-09-22T13:17:58.556 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:58.556 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bucket check olh --bucket check-bucket --dump-keys 2023-09-22T13:17:58.556 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:58.556 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:58.556 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.460+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.556 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.462+0000 7f655ba14500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.556 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.462+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.556 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.463+0000 7f655ba14500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.556 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.463+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.556 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.463+0000 7f655ba14500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.557 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.463+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.557 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.464+0000 7f655ba14500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.557 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.473+0000 7f655ba14500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:58.557 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.473+0000 7f655ba14500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:58.557 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.473+0000 7f655ba14500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:58.557 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.473+0000 7f655ba14500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:58.557 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.473+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.557 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.473+0000 7f655ba14500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.557 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.473+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.557 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.473+0000 7f655ba14500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.557 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.473+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.557 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.474+0000 7f655ba14500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.557 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.474+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.558 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.474+0000 7f655ba14500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.558 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.474+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.729 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.474+0000 7f655ba14500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.729 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.474+0000 7f655ba14500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:58.729 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.474+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.729 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.474+0000 7f655ba14500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.729 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.474+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.730 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.475+0000 7f655ba14500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.730 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.475+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.730 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.475+0000 7f655ba14500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.730 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.475+0000 7f655ba14500 20 zonegroup default 2023-09-22T13:17:58.730 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.475+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.730 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.475+0000 7f655ba14500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.730 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.475+0000 7f655ba14500 10 Cannot find current period zone using local zone 2023-09-22T13:17:58.730 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.475+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.730 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.475+0000 7f655ba14500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.730 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.475+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.730 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.476+0000 7f655ba14500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.731 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.476+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.731 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.476+0000 7f655ba14500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.731 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.476+0000 7f655ba14500 20 zone default found 2023-09-22T13:17:58.731 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.476+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.731 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.476+0000 7f655ba14500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.731 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.476+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.731 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.477+0000 7f655ba14500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.731 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.477+0000 7f655ba14500 20 started sync module instance, tier type = 2023-09-22T13:17:58.731 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.477+0000 7f655ba14500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:58.731 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.483+0000 7f6546ef6700 0 -- 172.21.15.96:0/2399193529 >> [v2:172.21.15.96:6824/140822,v1:172.21.15.96:6825/140822] conn(0x55dd6909f7c0 msgr2=0x55dd690a1c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:58.731 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.483+0000 7f655ba14500 20 add_watcher() i=0 2023-09-22T13:17:58.732 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.483+0000 7f655ba14500 20 add_watcher() i=1 2023-09-22T13:17:58.732 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.483+0000 7f655ba14500 20 add_watcher() i=2 2023-09-22T13:17:58.732 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.483+0000 7f655ba14500 20 add_watcher() i=3 2023-09-22T13:17:58.732 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.483+0000 7f655ba14500 20 add_watcher() i=4 2023-09-22T13:17:58.732 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.483+0000 7f655ba14500 20 add_watcher() i=5 2023-09-22T13:17:58.732 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.483+0000 7f655ba14500 20 add_watcher() i=6 2023-09-22T13:17:58.732 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.484+0000 7f655ba14500 20 add_watcher() i=7 2023-09-22T13:17:58.732 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.484+0000 7f655ba14500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:58.732 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.485+0000 7f6517fe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.732 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.485+0000 7f655ba14500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:58.732 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.485+0000 7f655ba14500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:58.732 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.490+0000 7f65476f7700 0 -- 172.21.15.96:0/3317665927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55dd690e1b40 msgr2=0x55dd690f2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:58.733 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.492+0000 7f655ba14500 5 note: GC not initialized 2023-09-22T13:17:58.733 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.493+0000 7f6486ffd700 20 reqs_thread_entry: start 2023-09-22T13:17:58.733 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.493+0000 7f6484ff9700 20 reqs_thread_entry: start 2023-09-22T13:17:58.733 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.493+0000 7f655ba14500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:58.733 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.493+0000 7f655ba14500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:58.733 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.493+0000 7f6476ffd700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:23 2023 2023-09-22T13:17:58.733 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:58.733 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.493+0000 7f64767fc700 20 reqs_thread_entry: start 2023-09-22T13:17:58.733 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.493+0000 7f655ba14500 20 get_system_obj_state: rctx=0x7ffdc01827b8 obj=default.rgw.meta:root:check-bucket state=0x55dd690531d0 s->prefetch_data=0 2023-09-22T13:17:58.733 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.493+0000 7f655ba14500 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-09-22T13:17:58.733 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x16 2023-09-22T13:17:58.734 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:58.734 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:58.734 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.734 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 10 cache get: name=default.rgw.meta+root+check-bucket : type miss (requested=0x11, cached=0x16) 2023-09-22T13:17:58.734 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.734 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:58.734 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-09-22T13:17:58.734 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 10 moving default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:58.734 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 20 get_system_obj_state: rctx=0x7ffdc0182888 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55dd690531d0 s->prefetch_data=0 2023-09-22T13:17:58.734 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : miss 2023-09-22T13:17:58.734 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x16 2023-09-22T13:17:58.735 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:58.735 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:58.735 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:58.735 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.735 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.735 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.735 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : type miss (requested=0x13, cached=0x16) 2023-09-22T13:17:58.735 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.494+0000 7f655ba14500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.735 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.495+0000 7f655ba14500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.736 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.495+0000 7f655ba14500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x13 2023-09-22T13:17:58.736 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.495+0000 7f655ba14500 10 moving default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:58.736 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.495+0000 7f655ba14500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:58.736 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.495+0000 7f655ba14500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:58.736 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.495+0000 7f655ba14500 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 2023-09-22T13:17:58.736 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.495+0000 7f655ba14500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:58.736 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.495+0000 7f655ba14500 1 NOTICE: finished shard 0 (0 entries found) 2023-09-22T13:17:58.736 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.495+0000 7f655ba14500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:58.736 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.496+0000 7f655ba14500 1 NOTICE: finished shard 1 (0 entries found) 2023-09-22T13:17:58.736 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.496+0000 7f655ba14500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:58.736 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.497+0000 7f655ba14500 1 NOTICE: finished shard 2 (0 entries found) 2023-09-22T13:17:58.737 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.497+0000 7f655ba14500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:58.737 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.497+0000 7f655ba14500 1 NOTICE: finished shard 3 (0 entries found) 2023-09-22T13:17:58.737 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.497+0000 7f655ba14500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:58.737 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.497+0000 7f655ba14500 1 NOTICE: finished shard 4 (0 entries found) 2023-09-22T13:17:58.737 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.497+0000 7f655ba14500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.737 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.498+0000 7f655ba14500 1 NOTICE: finished shard 5 (1 entries found) 2023-09-22T13:17:58.737 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.498+0000 7f655ba14500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:58.737 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.498+0000 7f655ba14500 1 NOTICE: finished shard 6 (0 entries found) 2023-09-22T13:17:58.737 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.498+0000 7f655ba14500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:58.737 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.499+0000 7f655ba14500 1 NOTICE: finished shard 7 (0 entries found) 2023-09-22T13:17:58.737 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.499+0000 7f655ba14500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.737 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.499+0000 7f655ba14500 1 NOTICE: finished shard 8 (1 entries found) 2023-09-22T13:17:58.738 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.499+0000 7f655ba14500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:58.738 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.500+0000 7f655ba14500 1 NOTICE: finished shard 9 (0 entries found) 2023-09-22T13:17:58.738 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.500+0000 7f655ba14500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:58.738 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.500+0000 7f655ba14500 1 NOTICE: finished shard 10 (0 entries found) 2023-09-22T13:17:58.738 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.500+0000 7f655ba14500 1 NOTICE: finished all shards (2 entries found) 2023-09-22T13:17:58.738 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.501+0000 7f655ba14500 20 remove_watcher() i=0 2023-09-22T13:17:58.738 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.501+0000 7f655ba14500 2 removed watcher, disabling cache 2023-09-22T13:17:58.738 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.502+0000 7f655ba14500 20 remove_watcher() i=1 2023-09-22T13:17:58.738 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.503+0000 7f655ba14500 20 remove_watcher() i=2 2023-09-22T13:17:58.738 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.503+0000 7f655ba14500 20 remove_watcher() i=3 2023-09-22T13:17:58.738 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.504+0000 7f655ba14500 20 remove_watcher() i=4 2023-09-22T13:17:58.738 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.505+0000 7f655ba14500 20 remove_watcher() i=5 2023-09-22T13:17:58.739 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.506+0000 7f655ba14500 20 remove_watcher() i=6 2023-09-22T13:17:58.739 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.506+0000 7f655ba14500 20 remove_watcher() i=7 2023-09-22T13:17:58.739 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.507+0000 7f6517fe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.739 INFO:tasks.workunit.client.0.smithi096.stderr:TEST: bucket check olh fixes leftover OLH entries 2023-09-22T13:17:58.739 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:58.739 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bucket check olh --bucket check-bucket --fix --rgw-olh-pending-timeout-sec 0 --dump-keys 2023-09-22T13:17:58.739 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:58.739 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:58.739 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.547+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.739 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.549+0000 7f712260e500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.739 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.549+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.739 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.550+0000 7f712260e500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.740 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.550+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.740 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.550+0000 7f712260e500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.740 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.550+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.740 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.551+0000 7f712260e500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.740 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.560+0000 7f712260e500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:58.740 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.560+0000 7f712260e500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:58.740 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.560+0000 7f712260e500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:58.740 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.560+0000 7f712260e500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:58.740 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.560+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.740 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.561+0000 7f712260e500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.740 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.561+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.740 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.561+0000 7f712260e500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.740 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.561+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.741 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.561+0000 7f712260e500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.741 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.561+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.741 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.561+0000 7f712260e500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.741 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.561+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.741 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.562+0000 7f712260e500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.741 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.562+0000 7f712260e500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:58.741 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.562+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.741 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.562+0000 7f712260e500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.741 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.562+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.741 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.562+0000 7f712260e500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.741 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.562+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.741 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.562+0000 7f712260e500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.742 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.562+0000 7f712260e500 20 zonegroup default 2023-09-22T13:17:58.742 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.562+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.742 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.563+0000 7f712260e500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.742 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.563+0000 7f712260e500 10 Cannot find current period zone using local zone 2023-09-22T13:17:58.742 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.563+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.742 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.563+0000 7f712260e500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.742 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.563+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.742 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.563+0000 7f712260e500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.742 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.563+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.742 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.563+0000 7f712260e500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.742 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.563+0000 7f712260e500 20 zone default found 2023-09-22T13:17:58.743 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.563+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.743 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.564+0000 7f712260e500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.743 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.564+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.743 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.564+0000 7f712260e500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.743 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.564+0000 7f712260e500 20 started sync module instance, tier type = 2023-09-22T13:17:58.743 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.564+0000 7f712260e500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:58.743 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.570+0000 7f712260e500 20 add_watcher() i=0 2023-09-22T13:17:58.743 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.570+0000 7f712260e500 20 add_watcher() i=1 2023-09-22T13:17:58.743 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.570+0000 7f712260e500 20 add_watcher() i=2 2023-09-22T13:17:58.743 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.570+0000 7f712260e500 20 add_watcher() i=3 2023-09-22T13:17:58.743 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.570+0000 7f712260e500 20 add_watcher() i=4 2023-09-22T13:17:58.743 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.570+0000 7f712260e500 20 add_watcher() i=5 2023-09-22T13:17:58.744 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.570+0000 7f712260e500 20 add_watcher() i=6 2023-09-22T13:17:58.744 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.571+0000 7f712260e500 20 add_watcher() i=7 2023-09-22T13:17:58.744 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.571+0000 7f712260e500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:58.744 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.571+0000 7f710d2ef700 0 -- 172.21.15.96:0/2787640784 >> [v2:172.21.15.96:6816/140810,v1:172.21.15.96:6817/140810] conn(0x55e685f73cd0 msgr2=0x55e685f760c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:58.744 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.573+0000 7f70df7de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.744 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.573+0000 7f712260e500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:58.744 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.573+0000 7f712260e500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:58.744 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.577+0000 7f710e2f1700 0 -- 172.21.15.96:0/1047006437 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55e685fb9d10 msgr2=0x55e685fbdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:58.744 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.579+0000 7f712260e500 5 note: GC not initialized 2023-09-22T13:17:58.744 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.580+0000 7f704e7fc700 20 reqs_thread_entry: start 2023-09-22T13:17:58.744 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.580+0000 7f703ffff700 20 reqs_thread_entry: start 2023-09-22T13:17:58.745 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.580+0000 7f712260e500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:58.745 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.580+0000 7f712260e500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:58.745 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.581+0000 7f703e7fc700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:24 2023 2023-09-22T13:17:58.745 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:58.745 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.581+0000 7f703dffb700 20 reqs_thread_entry: start 2023-09-22T13:17:58.745 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.581+0000 7f712260e500 20 get_system_obj_state: rctx=0x7fff534a5958 obj=default.rgw.meta:root:check-bucket state=0x55e685fc0560 s->prefetch_data=0 2023-09-22T13:17:58.745 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.581+0000 7f712260e500 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-09-22T13:17:58.745 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.581+0000 7f712260e500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x16 2023-09-22T13:17:58.745 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.581+0000 7f712260e500 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:58.745 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.581+0000 7f712260e500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:58.745 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.581+0000 7f712260e500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.745 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.581+0000 7f712260e500 10 cache get: name=default.rgw.meta+root+check-bucket : type miss (requested=0x11, cached=0x16) 2023-09-22T13:17:58.746 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.581+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.746 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.581+0000 7f712260e500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:58.746 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.581+0000 7f712260e500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-09-22T13:17:58.746 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.581+0000 7f712260e500 10 moving default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:58.746 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.581+0000 7f712260e500 20 get_system_obj_state: rctx=0x7fff534a5a28 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55e685fc0560 s->prefetch_data=0 2023-09-22T13:17:58.746 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.581+0000 7f712260e500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : miss 2023-09-22T13:17:58.746 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.582+0000 7f712260e500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x16 2023-09-22T13:17:58.746 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.582+0000 7f712260e500 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:58.746 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.582+0000 7f712260e500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:58.746 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.582+0000 7f712260e500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:58.746 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.582+0000 7f712260e500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.747 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.582+0000 7f712260e500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.747 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.582+0000 7f712260e500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.747 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.582+0000 7f712260e500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : type miss (requested=0x13, cached=0x16) 2023-09-22T13:17:58.747 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.582+0000 7f712260e500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.747 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.582+0000 7f712260e500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.747 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.582+0000 7f712260e500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x13 2023-09-22T13:17:58.747 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.582+0000 7f712260e500 10 moving default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:58.747 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.582+0000 7f712260e500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:58.747 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.582+0000 7f712260e500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:58.747 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.582+0000 7f712260e500 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 2023-09-22T13:17:58.747 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.582+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:58.747 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.583+0000 7f712260e500 1 NOTICE: finished shard 0 (0 entries removed) 2023-09-22T13:17:58.748 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.583+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:58.748 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.583+0000 7f712260e500 1 NOTICE: finished shard 1 (0 entries removed) 2023-09-22T13:17:58.748 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.583+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:58.748 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.584+0000 7f712260e500 1 NOTICE: finished shard 2 (0 entries removed) 2023-09-22T13:17:58.748 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.584+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:58.748 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.584+0000 7f712260e500 1 NOTICE: finished shard 3 (0 entries removed) 2023-09-22T13:17:58.748 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.584+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:58.748 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.585+0000 7f712260e500 1 NOTICE: finished shard 4 (0 entries removed) 2023-09-22T13:17:58.748 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.585+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.749 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.585+0000 7f712260e500 20 get_obj_state: rctx=0x55e686052b80 obj=check-bucket:d state=0x55e686055e38 s->prefetch_data=0 2023-09-22T13:17:58.749 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.585+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:58.749 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.586+0000 7f712260e500 1 NOTICE: finished shard 6 (0 entries removed) 2023-09-22T13:17:58.749 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.586+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:58.749 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.586+0000 7f712260e500 1 NOTICE: finished shard 7 (0 entries removed) 2023-09-22T13:17:58.749 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.586+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.749 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.587+0000 7f712260e500 20 get_obj_state: rctx=0x55e686076980 obj=check-bucket:c state=0x55e686077398 s->prefetch_data=0 2023-09-22T13:17:58.749 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.587+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:58.749 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.587+0000 7f712260e500 1 NOTICE: finished shard 9 (0 entries removed) 2023-09-22T13:17:58.749 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.587+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:58.749 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.587+0000 7f712260e500 1 NOTICE: finished shard 10 (0 entries removed) 2023-09-22T13:17:58.750 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.588+0000 7f712260e500 20 get_obj_state: setting s->obj_tag to 5ae0tg9nugofps6h1wwyb1pqu0favgeh 2023-09-22T13:17:58.750 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.588+0000 7f712260e500 20 get_obj_state_impl: setting s->olh_tag to 7qsd3679ddkkjzkav8aiagm75tlvzsq5 2023-09-22T13:17:58.750 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.588+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.750 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.588+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.750 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.588+0000 7f712260e500 20 olh_log_entry: epoch=4 op=2 key=d[] 2023-09-22T13:17:58.750 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.588+0000 7f712260e500 20 olh_log_entry: epoch=4 op=3 key=d[TXjBqOO2nYj05OnheLExQmVWjEwUWgE] 2023-09-22T13:17:58.750 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.588+0000 7f712260e500 20 get_obj_state: rctx=0x55e686052b80 obj=check-bucket:_:TXjBqOO2nYj05OnheLExQmVWjEwUWgE_d state=0x55e68607ce08 s->prefetch_data=0 2023-09-22T13:17:58.750 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.589+0000 7f712260e500 20 get_obj_state: rctx=0x55e686052b80 obj=check-bucket:d state=0x55e686055e38 s->prefetch_data=0 2023-09-22T13:17:58.750 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.589+0000 7f712260e500 20 bucket_index_guard_olh_op(): olh_state.olh_tag=7qsd3679ddkkjzkav8aiagm75tlvzsq5 2023-09-22T13:17:58.750 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.590+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.750 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.591+0000 7f712260e500 1 NOTICE: finished shard 5 (1 entries removed) 2023-09-22T13:17:58.751 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.591+0000 7f712260e500 20 get_obj_state: setting s->obj_tag to jogkk1lyeocreujbycjtrtll8bgivcxm 2023-09-22T13:17:58.751 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.591+0000 7f712260e500 20 get_obj_state_impl: setting s->olh_tag to pww24tiki3p98t3vt06ip99sghububrl 2023-09-22T13:17:58.751 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.591+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.751 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.591+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.751 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.592+0000 7f712260e500 20 olh_log_entry: epoch=4 op=2 key=c[] 2023-09-22T13:17:58.751 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.592+0000 7f712260e500 20 olh_log_entry: epoch=4 op=3 key=c[fwcEb-h3AuJ-YOkdgSkxKZMLz0df6tl] 2023-09-22T13:17:58.751 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.592+0000 7f712260e500 20 get_obj_state: rctx=0x55e686076980 obj=check-bucket:_:fwcEb-h3AuJ-YOkdgSkxKZMLz0df6tl_c state=0x55e686055e38 s->prefetch_data=0 2023-09-22T13:17:58.751 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.593+0000 7f712260e500 20 get_obj_state: rctx=0x55e686076980 obj=check-bucket:c state=0x55e686077398 s->prefetch_data=0 2023-09-22T13:17:58.751 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.593+0000 7f712260e500 20 bucket_index_guard_olh_op(): olh_state.olh_tag=pww24tiki3p98t3vt06ip99sghububrl 2023-09-22T13:17:58.751 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.594+0000 7f712260e500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.751 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.596+0000 7f712260e500 1 NOTICE: finished shard 8 (1 entries removed) 2023-09-22T13:17:58.751 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.596+0000 7f712260e500 1 NOTICE: finished all shards (2 entries removed) 2023-09-22T13:17:58.752 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.597+0000 7f712260e500 20 remove_watcher() i=0 2023-09-22T13:17:58.752 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.597+0000 7f712260e500 2 removed watcher, disabling cache 2023-09-22T13:17:58.752 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.598+0000 7f712260e500 20 remove_watcher() i=1 2023-09-22T13:17:58.752 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.598+0000 7f712260e500 20 remove_watcher() i=2 2023-09-22T13:17:58.752 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.599+0000 7f712260e500 20 remove_watcher() i=3 2023-09-22T13:17:58.752 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.600+0000 7f712260e500 20 remove_watcher() i=4 2023-09-22T13:17:58.752 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.601+0000 7f712260e500 20 remove_watcher() i=5 2023-09-22T13:17:58.752 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.601+0000 7f712260e500 20 remove_watcher() i=6 2023-09-22T13:17:58.752 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.602+0000 7f712260e500 20 remove_watcher() i=7 2023-09-22T13:17:58.752 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.603+0000 7f70df7de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.752 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bi list --bucket check-bucket --object c 2023-09-22T13:17:58.752 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:58.753 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:58.753 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.642+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.753 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.644+0000 7fb734603500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.753 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.644+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.753 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.644+0000 7fb734603500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.753 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.644+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.753 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.645+0000 7fb734603500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.753 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.645+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.753 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.645+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.753 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.655+0000 7fb734603500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:58.753 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.655+0000 7fb734603500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:58.753 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.655+0000 7fb734603500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:58.754 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.655+0000 7fb734603500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:58.754 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.655+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.754 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.656+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.754 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.656+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.754 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.656+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.754 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.656+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.754 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.656+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.754 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.656+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.754 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.656+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.754 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.656+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.754 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.657+0000 7fb734603500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.754 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.657+0000 7fb734603500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:58.755 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.657+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.755 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.657+0000 7fb734603500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.755 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.657+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.755 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.657+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.755 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.657+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.755 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.657+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.755 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.657+0000 7fb734603500 20 zonegroup default 2023-09-22T13:17:58.755 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.657+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.755 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.657+0000 7fb734603500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.755 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.657+0000 7fb734603500 10 Cannot find current period zone using local zone 2023-09-22T13:17:58.883 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.657+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.883 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.658+0000 7fb734603500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.883 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.658+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.884 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.658+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.884 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.658+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.884 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.658+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.884 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.658+0000 7fb734603500 20 zone default found 2023-09-22T13:17:58.884 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.658+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.884 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.659+0000 7fb734603500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.884 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.659+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.884 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.659+0000 7fb734603500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.884 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.659+0000 7fb734603500 20 started sync module instance, tier type = 2023-09-22T13:17:58.885 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.659+0000 7fb734603500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:58.885 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.665+0000 7fb734603500 20 add_watcher() i=0 2023-09-22T13:17:58.885 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.665+0000 7fb734603500 20 add_watcher() i=1 2023-09-22T13:17:58.885 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.665+0000 7fb734603500 20 add_watcher() i=2 2023-09-22T13:17:58.885 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.665+0000 7fb734603500 20 add_watcher() i=3 2023-09-22T13:17:58.885 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.665+0000 7fb734603500 20 add_watcher() i=4 2023-09-22T13:17:58.885 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.665+0000 7fb734603500 20 add_watcher() i=5 2023-09-22T13:17:58.885 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.665+0000 7fb734603500 20 add_watcher() i=6 2023-09-22T13:17:58.885 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.666+0000 7fb734603500 20 add_watcher() i=7 2023-09-22T13:17:58.885 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.666+0000 7fb734603500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:58.885 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.666+0000 7fb7202e6700 0 -- 172.21.15.96:0/4263339131 >> [v2:172.21.15.96:6816/140810,v1:172.21.15.96:6817/140810] conn(0x5601afddc4e0 msgr2=0x5601afdde8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-09-22T13:17:58.885 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.668+0000 7fb6f17e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.886 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.668+0000 7fb734603500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:58.886 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.668+0000 7fb734603500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:58.886 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.672+0000 7fb71f2e4700 0 -- 172.21.15.96:0/3033952659 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5601afe233b0 msgr2=0x5601afe18d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-09-22T13:17:58.886 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.675+0000 7fb734603500 5 note: GC not initialized 2023-09-22T13:17:58.886 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.676+0000 7fb65ffff700 20 reqs_thread_entry: start 2023-09-22T13:17:58.886 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.676+0000 7fb65dffb700 20 reqs_thread_entry: start 2023-09-22T13:17:58.886 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.676+0000 7fb734603500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:58.886 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.676+0000 7fb734603500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:58.886 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.676+0000 7fb64bfff700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:23 2023 2023-09-22T13:17:58.886 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:58.886 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.676+0000 7fb64b7fe700 20 reqs_thread_entry: start 2023-09-22T13:17:58.887 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.676+0000 7fb734603500 20 get_system_obj_state: rctx=0x7fff98cd59b0 obj=default.rgw.meta:root:check-bucket state=0x5601afe204d0 s->prefetch_data=0 2023-09-22T13:17:58.887 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.677+0000 7fb734603500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.887 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.677+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.887 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.677+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:58.887 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.677+0000 7fb734603500 20 get_system_obj_state: rctx=0x7fff98cd59b0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5601afdde2c0 s->prefetch_data=0 2023-09-22T13:17:58.887 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.677+0000 7fb734603500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.887 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.677+0000 7fb734603500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.887 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.677+0000 7fb734603500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.887 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.677+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.887 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.678+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.887 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.678+0000 7fb734603500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.888 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.678+0000 7fb734603500 20 get_system_obj_state: rctx=0x7fff98cd5780 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5601afdde2c0 s->prefetch_data=0 2023-09-22T13:17:58.888 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.678+0000 7fb734603500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.888 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.678+0000 7fb734603500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.888 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.678+0000 7fb734603500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.888 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.678+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.888 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.679+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.888 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.679+0000 7fb734603500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.888 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.679+0000 7fb734603500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:58.888 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.679+0000 7fb734603500 20 get_system_obj_state: rctx=0x7fff98cd5780 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5601afdde2c0 s->prefetch_data=0 2023-09-22T13:17:58.888 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.680+0000 7fb734603500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.889 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.680+0000 7fb734603500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.889 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.680+0000 7fb734603500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.889 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.680+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.889 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.680+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.889 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.680+0000 7fb734603500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.889 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.680+0000 7fb734603500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:58.889 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.681+0000 7fb734603500 20 get_system_obj_state: rctx=0x7fff98cd5780 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5601afdde2c0 s->prefetch_data=0 2023-09-22T13:17:58.889 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.681+0000 7fb734603500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.889 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.681+0000 7fb734603500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.889 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.681+0000 7fb734603500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.889 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.681+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.889 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.681+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.890 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.681+0000 7fb734603500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.890 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.681+0000 7fb734603500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:58.890 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.682+0000 7fb734603500 20 get_system_obj_state: rctx=0x7fff98cd5780 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5601afdde2c0 s->prefetch_data=0 2023-09-22T13:17:58.890 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.682+0000 7fb734603500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.890 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.682+0000 7fb734603500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.890 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.682+0000 7fb734603500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.890 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.682+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.890 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.683+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.890 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.683+0000 7fb734603500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.890 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.683+0000 7fb734603500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:58.890 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.683+0000 7fb734603500 20 get_system_obj_state: rctx=0x7fff98cd5780 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5601afdde2c0 s->prefetch_data=0 2023-09-22T13:17:58.891 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.684+0000 7fb734603500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.891 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.684+0000 7fb734603500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.891 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.684+0000 7fb734603500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.891 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.684+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.891 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.684+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.891 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.684+0000 7fb734603500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.891 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.684+0000 7fb734603500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:58.891 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.684+0000 7fb734603500 20 get_system_obj_state: rctx=0x7fff98cd5780 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5601afdde2c0 s->prefetch_data=0 2023-09-22T13:17:58.891 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.685+0000 7fb734603500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.891 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.685+0000 7fb734603500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.891 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.685+0000 7fb734603500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.892 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.685+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.892 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.685+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.892 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.685+0000 7fb734603500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.892 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.685+0000 7fb734603500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.892 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.686+0000 7fb734603500 20 get_system_obj_state: rctx=0x7fff98cd5780 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5601afdde2c0 s->prefetch_data=0 2023-09-22T13:17:58.892 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.686+0000 7fb734603500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.892 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.686+0000 7fb734603500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.892 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.686+0000 7fb734603500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.892 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.686+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.892 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.686+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.892 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.686+0000 7fb734603500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.892 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.686+0000 7fb734603500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:58.893 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.687+0000 7fb734603500 20 get_system_obj_state: rctx=0x7fff98cd5780 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5601afdde2c0 s->prefetch_data=0 2023-09-22T13:17:58.893 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.687+0000 7fb734603500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.893 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.687+0000 7fb734603500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.893 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.687+0000 7fb734603500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.893 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.687+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.893 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.688+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.893 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.688+0000 7fb734603500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.893 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.688+0000 7fb734603500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:58.893 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.688+0000 7fb734603500 20 get_system_obj_state: rctx=0x7fff98cd5780 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5601afdde2c0 s->prefetch_data=0 2023-09-22T13:17:58.893 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.688+0000 7fb734603500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.893 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.688+0000 7fb734603500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.894 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.688+0000 7fb734603500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.894 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.688+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.894 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.689+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.894 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.689+0000 7fb734603500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.894 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.689+0000 7fb734603500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.894 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.689+0000 7fb734603500 20 get_system_obj_state: rctx=0x7fff98cd5780 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5601afdde2c0 s->prefetch_data=0 2023-09-22T13:17:58.894 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.690+0000 7fb734603500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.894 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.690+0000 7fb734603500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.894 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.690+0000 7fb734603500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.894 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.690+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.894 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.690+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.895 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.690+0000 7fb734603500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.895 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.690+0000 7fb734603500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:58.895 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.690+0000 7fb734603500 20 get_system_obj_state: rctx=0x7fff98cd5780 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5601afdde2c0 s->prefetch_data=0 2023-09-22T13:17:58.895 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.691+0000 7fb734603500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.895 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.691+0000 7fb734603500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.895 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.691+0000 7fb734603500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.895 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.691+0000 7fb734603500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.895 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.691+0000 7fb734603500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.895 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.691+0000 7fb734603500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.895 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.691+0000 7fb734603500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:58.895 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.693+0000 7fb734603500 20 remove_watcher() i=0 2023-09-22T13:17:58.895 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.693+0000 7fb734603500 2 removed watcher, disabling cache 2023-09-22T13:17:58.896 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.694+0000 7fb734603500 20 remove_watcher() i=1 2023-09-22T13:17:58.896 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.694+0000 7fb734603500 20 remove_watcher() i=2 2023-09-22T13:17:58.896 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.695+0000 7fb734603500 20 remove_watcher() i=3 2023-09-22T13:17:58.896 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.696+0000 7fb734603500 20 remove_watcher() i=4 2023-09-22T13:17:58.896 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.697+0000 7fb734603500 20 remove_watcher() i=5 2023-09-22T13:17:58.896 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.697+0000 7fb734603500 20 remove_watcher() i=6 2023-09-22T13:17:58.896 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.698+0000 7fb734603500 20 remove_watcher() i=7 2023-09-22T13:17:58.896 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.699+0000 7fb6f17e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.896 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bi list --bucket check-bucket --object d 2023-09-22T13:17:58.896 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:58.896 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:58.896 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.737+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.896 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.738+0000 7fa880d9c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.897 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.738+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.897 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.738+0000 7fa880d9c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.897 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.738+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.897 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.739+0000 7fa880d9c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.897 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.739+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.897 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.740+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.897 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.749+0000 7fa880d9c500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:58.897 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.749+0000 7fa880d9c500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:58.897 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.749+0000 7fa880d9c500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:58.897 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.749+0000 7fa880d9c500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:58.897 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.749+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.897 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.749+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.897 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.749+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.898 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.750+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.898 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.750+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.898 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.750+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.898 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.750+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.898 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.750+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.898 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.750+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.898 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.751+0000 7fa880d9c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.898 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.751+0000 7fa880d9c500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:58.898 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.751+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.898 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.751+0000 7fa880d9c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.898 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.751+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.899 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.751+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.899 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.751+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.899 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.751+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:58.899 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.751+0000 7fa880d9c500 20 zonegroup default 2023-09-22T13:17:58.899 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.751+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.899 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.751+0000 7fa880d9c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.899 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.752+0000 7fa880d9c500 10 Cannot find current period zone using local zone 2023-09-22T13:17:58.899 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.752+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.899 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.752+0000 7fa880d9c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.899 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.752+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.899 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.752+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:58.899 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.752+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.900 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.752+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:58.900 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.752+0000 7fa880d9c500 20 zone default found 2023-09-22T13:17:58.900 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.752+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.900 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.752+0000 7fa880d9c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.900 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.753+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.900 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.753+0000 7fa880d9c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:58.900 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.753+0000 7fa880d9c500 20 started sync module instance, tier type = 2023-09-22T13:17:58.900 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.753+0000 7fa880d9c500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:58.900 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.759+0000 7fa86c27e700 0 -- 172.21.15.96:0/3247161317 >> [v2:172.21.15.96:6824/140822,v1:172.21.15.96:6825/140822] conn(0x557f36a18950 msgr2=0x557f36a1e9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:58.900 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.759+0000 7fa880d9c500 20 add_watcher() i=0 2023-09-22T13:17:58.900 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.759+0000 7fa880d9c500 20 add_watcher() i=1 2023-09-22T13:17:58.900 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.759+0000 7fa880d9c500 20 add_watcher() i=2 2023-09-22T13:17:58.901 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.759+0000 7fa880d9c500 20 add_watcher() i=3 2023-09-22T13:17:58.901 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.759+0000 7fa880d9c500 20 add_watcher() i=4 2023-09-22T13:17:58.901 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.759+0000 7fa880d9c500 20 add_watcher() i=5 2023-09-22T13:17:58.901 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.759+0000 7fa880d9c500 20 add_watcher() i=6 2023-09-22T13:17:58.901 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.760+0000 7fa880d9c500 20 add_watcher() i=7 2023-09-22T13:17:58.901 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.760+0000 7fa880d9c500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:58.901 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.761+0000 7fa83dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:58.901 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.762+0000 7fa880d9c500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:58.901 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.762+0000 7fa880d9c500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:58.901 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.765+0000 7fa86ba7d700 0 -- 172.21.15.96:0/406932460 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557f36a61020 msgr2=0x557f36a72a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-09-22T13:17:58.901 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.767+0000 7fa880d9c500 5 note: GC not initialized 2023-09-22T13:17:58.901 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.768+0000 7fa7b4ff9700 20 reqs_thread_entry: start 2023-09-22T13:17:58.901 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.768+0000 7fa7aa7fc700 20 reqs_thread_entry: start 2023-09-22T13:17:58.902 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.768+0000 7fa880d9c500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:58.902 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.768+0000 7fa880d9c500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:58.902 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.768+0000 7fa7a8ff9700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:23 2023 2023-09-22T13:17:58.902 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:58.902 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.769+0000 7fa78ffff700 20 reqs_thread_entry: start 2023-09-22T13:17:58.902 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.769+0000 7fa880d9c500 20 get_system_obj_state: rctx=0x7ffe1303ded0 obj=default.rgw.meta:root:check-bucket state=0x557f368ad810 s->prefetch_data=0 2023-09-22T13:17:58.902 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.769+0000 7fa880d9c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.902 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.769+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.902 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.769+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:58.902 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.769+0000 7fa880d9c500 20 get_system_obj_state: rctx=0x7ffe1303ded0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x557f36867910 s->prefetch_data=0 2023-09-22T13:17:58.902 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.770+0000 7fa880d9c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.903 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.770+0000 7fa880d9c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.903 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.770+0000 7fa880d9c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.903 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.770+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.903 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.770+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.903 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.770+0000 7fa880d9c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.903 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.770+0000 7fa880d9c500 20 get_system_obj_state: rctx=0x7ffe1303dca0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x557f36867910 s->prefetch_data=0 2023-09-22T13:17:58.903 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.770+0000 7fa880d9c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.903 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.770+0000 7fa880d9c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.903 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.770+0000 7fa880d9c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.903 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.770+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.903 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.770+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.903 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.771+0000 7fa880d9c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.903 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.771+0000 7fa880d9c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:58.904 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.771+0000 7fa880d9c500 20 get_system_obj_state: rctx=0x7ffe1303dca0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x557f36867910 s->prefetch_data=0 2023-09-22T13:17:58.904 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.771+0000 7fa880d9c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.904 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.771+0000 7fa880d9c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.904 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.771+0000 7fa880d9c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.904 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.771+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.904 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.772+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.904 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.772+0000 7fa880d9c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.904 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.772+0000 7fa880d9c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:58.904 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.772+0000 7fa880d9c500 20 get_system_obj_state: rctx=0x7ffe1303dca0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x557f36867910 s->prefetch_data=0 2023-09-22T13:17:58.904 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.773+0000 7fa880d9c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.904 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.773+0000 7fa880d9c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.905 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.773+0000 7fa880d9c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.905 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.773+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.905 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.773+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.905 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.773+0000 7fa880d9c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.905 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.773+0000 7fa880d9c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:58.905 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.773+0000 7fa880d9c500 20 get_system_obj_state: rctx=0x7ffe1303dca0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x557f36867910 s->prefetch_data=0 2023-09-22T13:17:58.905 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.774+0000 7fa880d9c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.905 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.774+0000 7fa880d9c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.905 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.774+0000 7fa880d9c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.905 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.774+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.905 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.774+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.905 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.774+0000 7fa880d9c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.905 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.774+0000 7fa880d9c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:58.906 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.775+0000 7fa880d9c500 20 get_system_obj_state: rctx=0x7ffe1303dca0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x557f36867910 s->prefetch_data=0 2023-09-22T13:17:58.906 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.775+0000 7fa880d9c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.906 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.775+0000 7fa880d9c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.906 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.775+0000 7fa880d9c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.906 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.775+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.906 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.775+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.906 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.775+0000 7fa880d9c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.906 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.775+0000 7fa880d9c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:58.906 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.776+0000 7fa880d9c500 20 get_system_obj_state: rctx=0x7ffe1303dca0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x557f36867910 s->prefetch_data=0 2023-09-22T13:17:58.906 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.776+0000 7fa880d9c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.906 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.776+0000 7fa880d9c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.906 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.776+0000 7fa880d9c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.907 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.776+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.907 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.776+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.907 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.776+0000 7fa880d9c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.907 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.776+0000 7fa880d9c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:58.907 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.777+0000 7fa880d9c500 20 get_system_obj_state: rctx=0x7ffe1303dca0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x557f36867910 s->prefetch_data=0 2023-09-22T13:17:58.907 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.777+0000 7fa880d9c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.907 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.777+0000 7fa880d9c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.907 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.777+0000 7fa880d9c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.907 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.777+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.907 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.778+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.908 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.778+0000 7fa880d9c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.908 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.778+0000 7fa880d9c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:58.908 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.778+0000 7fa880d9c500 20 get_system_obj_state: rctx=0x7ffe1303dca0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x557f36867910 s->prefetch_data=0 2023-09-22T13:17:58.908 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.779+0000 7fa880d9c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.908 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.779+0000 7fa880d9c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.908 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.779+0000 7fa880d9c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.908 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.779+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.908 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.779+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.908 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.779+0000 7fa880d9c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.908 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.779+0000 7fa880d9c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:58.908 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.780+0000 7fa880d9c500 20 get_system_obj_state: rctx=0x7ffe1303dca0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x557f36867910 s->prefetch_data=0 2023-09-22T13:17:58.909 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.780+0000 7fa880d9c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.909 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.780+0000 7fa880d9c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.909 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.780+0000 7fa880d9c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.909 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.780+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.909 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.780+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.909 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.780+0000 7fa880d9c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.909 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.780+0000 7fa880d9c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:58.909 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.781+0000 7fa880d9c500 20 get_system_obj_state: rctx=0x7ffe1303dca0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x557f36867910 s->prefetch_data=0 2023-09-22T13:17:58.909 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.781+0000 7fa880d9c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.909 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.781+0000 7fa880d9c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.909 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.781+0000 7fa880d9c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.909 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.781+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.910 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.781+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.910 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.781+0000 7fa880d9c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.910 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.781+0000 7fa880d9c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:58.910 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.782+0000 7fa880d9c500 20 get_system_obj_state: rctx=0x7ffe1303dca0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x557f36867910 s->prefetch_data=0 2023-09-22T13:17:58.910 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.782+0000 7fa880d9c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:58.910 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.782+0000 7fa880d9c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:58.910 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.782+0000 7fa880d9c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:58.910 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.782+0000 7fa880d9c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:58.910 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.782+0000 7fa880d9c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:58.910 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.782+0000 7fa880d9c500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:58.911 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.782+0000 7fa880d9c500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:58.911 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.784+0000 7fa880d9c500 20 remove_watcher() i=0 2023-09-22T13:17:58.911 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.784+0000 7fa880d9c500 2 removed watcher, disabling cache 2023-09-22T13:17:58.911 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.784+0000 7fa880d9c500 20 remove_watcher() i=1 2023-09-22T13:17:59.036 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.785+0000 7fa880d9c500 20 remove_watcher() i=2 2023-09-22T13:17:59.036 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.786+0000 7fa880d9c500 20 remove_watcher() i=3 2023-09-22T13:17:59.036 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.787+0000 7fa880d9c500 20 remove_watcher() i=4 2023-09-22T13:17:59.036 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.788+0000 7fa880d9c500 20 remove_watcher() i=5 2023-09-22T13:17:59.037 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.788+0000 7fa880d9c500 20 remove_watcher() i=6 2023-09-22T13:17:59.037 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.789+0000 7fa880d9c500 20 remove_watcher() i=7 2023-09-22T13:17:59.037 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.790+0000 7fa83dfe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.037 INFO:tasks.workunit.client.0.smithi096.stderr:TEST: bucket check olh does not find new leftover OLH entries after fix 2023-09-22T13:17:59.037 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:59.037 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bucket check olh --bucket check-bucket --dump-keys 2023-09-22T13:17:59.037 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:59.037 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:59.037 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.828+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.037 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.829+0000 7fc46c1d5500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.037 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.829+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.038 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.830+0000 7fc46c1d5500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.038 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.830+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.038 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.830+0000 7fc46c1d5500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.038 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.830+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.038 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.831+0000 7fc46c1d5500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.038 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.840+0000 7fc46c1d5500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:59.038 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.840+0000 7fc46c1d5500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:59.038 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.840+0000 7fc46c1d5500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:59.038 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.840+0000 7fc46c1d5500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:59.038 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.840+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.039 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.840+0000 7fc46c1d5500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.039 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.840+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.039 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.841+0000 7fc46c1d5500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:59.039 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.841+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.039 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.841+0000 7fc46c1d5500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.039 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.841+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.039 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.841+0000 7fc46c1d5500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:59.039 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.841+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.039 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.841+0000 7fc46c1d5500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.039 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.841+0000 7fc46c1d5500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:59.039 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.841+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.039 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.841+0000 7fc46c1d5500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.040 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.841+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.040 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.842+0000 7fc46c1d5500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.040 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.842+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.040 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.842+0000 7fc46c1d5500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:59.040 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.842+0000 7fc46c1d5500 20 zonegroup default 2023-09-22T13:17:59.040 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.842+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.040 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.842+0000 7fc46c1d5500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.040 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.842+0000 7fc46c1d5500 10 Cannot find current period zone using local zone 2023-09-22T13:17:59.040 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.842+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.040 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.843+0000 7fc46c1d5500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.040 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.843+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.041 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.843+0000 7fc46c1d5500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.041 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.843+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.041 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.843+0000 7fc46c1d5500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:59.041 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.843+0000 7fc46c1d5500 20 zone default found 2023-09-22T13:17:59.041 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.843+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.041 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.843+0000 7fc46c1d5500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.041 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.843+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.041 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.844+0000 7fc46c1d5500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.041 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.844+0000 7fc46c1d5500 20 started sync module instance, tier type = 2023-09-22T13:17:59.041 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.844+0000 7fc46c1d5500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:59.041 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.850+0000 7fc4576b7700 0 -- 172.21.15.96:0/1441971649 >> [v2:172.21.15.96:6807/140797,v1:172.21.15.96:6809/140797] conn(0x7fc44c02f910 msgr2=0x7fc44c012390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:59.041 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.850+0000 7fc46c1d5500 20 add_watcher() i=0 2023-09-22T13:17:59.042 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.850+0000 7fc46c1d5500 20 add_watcher() i=1 2023-09-22T13:17:59.042 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.850+0000 7fc46c1d5500 20 add_watcher() i=2 2023-09-22T13:17:59.042 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.850+0000 7fc46c1d5500 20 add_watcher() i=3 2023-09-22T13:17:59.042 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.850+0000 7fc46c1d5500 20 add_watcher() i=4 2023-09-22T13:17:59.042 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.850+0000 7fc46c1d5500 20 add_watcher() i=5 2023-09-22T13:17:59.042 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.850+0000 7fc46c1d5500 20 add_watcher() i=6 2023-09-22T13:17:59.042 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.851+0000 7fc46c1d5500 20 add_watcher() i=7 2023-09-22T13:17:59.042 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.851+0000 7fc46c1d5500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:59.042 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.852+0000 7fc3a3fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.042 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.853+0000 7fc46c1d5500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:59.042 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.853+0000 7fc46c1d5500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:59.043 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:53.857+0000 7fc4576b7700 0 -- 172.21.15.96:0/3773516182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cd2eada30 msgr2=0x564cd2ec5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:59.043 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.058+0000 7fc4576b7700 0 --2- 172.21.15.96:0/3773516182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564cd2eada30 0x564cd2ec5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-09-22T13:17:59.043 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.060+0000 7fc46c1d5500 5 note: GC not initialized 2023-09-22T13:17:59.043 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.061+0000 7fc392ffd700 20 reqs_thread_entry: start 2023-09-22T13:17:59.043 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.061+0000 7fc390ff9700 20 reqs_thread_entry: start 2023-09-22T13:17:59.043 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.061+0000 7fc46c1d5500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:59.043 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.061+0000 7fc46c1d5500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:59.043 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.061+0000 7fc38affd700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:24 2023 2023-09-22T13:17:59.043 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:59.043 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.061+0000 7fc38a7fc700 20 reqs_thread_entry: start 2023-09-22T13:17:59.043 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.061+0000 7fc46c1d5500 20 get_system_obj_state: rctx=0x7ffcf8f23698 obj=default.rgw.meta:root:check-bucket state=0x564cd2e4dd20 s->prefetch_data=0 2023-09-22T13:17:59.044 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.061+0000 7fc46c1d5500 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-09-22T13:17:59.044 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x16 2023-09-22T13:17:59.044 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:59.044 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:59.044 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.044 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 10 cache get: name=default.rgw.meta+root+check-bucket : type miss (requested=0x11, cached=0x16) 2023-09-22T13:17:59.044 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.044 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:59.044 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-09-22T13:17:59.044 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 10 moving default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:59.044 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 20 get_system_obj_state: rctx=0x7ffcf8f23768 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x564cd2e4dd20 s->prefetch_data=0 2023-09-22T13:17:59.044 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : miss 2023-09-22T13:17:59.045 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x16 2023-09-22T13:17:59.045 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:59.045 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:59.045 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:59.045 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.045 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.045 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.045 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : type miss (requested=0x13, cached=0x16) 2023-09-22T13:17:59.045 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.062+0000 7fc46c1d5500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.045 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.063+0000 7fc46c1d5500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.045 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.063+0000 7fc46c1d5500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x13 2023-09-22T13:17:59.046 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.063+0000 7fc46c1d5500 10 moving default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:59.046 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.063+0000 7fc46c1d5500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:59.046 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.063+0000 7fc46c1d5500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:59.046 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.063+0000 7fc46c1d5500 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 2023-09-22T13:17:59.046 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.063+0000 7fc46c1d5500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:59.046 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.063+0000 7fc46c1d5500 1 NOTICE: finished shard 0 (0 entries found) 2023-09-22T13:17:59.046 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.063+0000 7fc46c1d5500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:59.046 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.064+0000 7fc46c1d5500 1 NOTICE: finished shard 1 (0 entries found) 2023-09-22T13:17:59.046 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.064+0000 7fc46c1d5500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:59.046 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.065+0000 7fc46c1d5500 1 NOTICE: finished shard 2 (0 entries found) 2023-09-22T13:17:59.046 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.065+0000 7fc46c1d5500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:59.047 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.065+0000 7fc46c1d5500 1 NOTICE: finished shard 3 (0 entries found) 2023-09-22T13:17:59.047 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.065+0000 7fc46c1d5500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:59.047 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.066+0000 7fc46c1d5500 1 NOTICE: finished shard 4 (0 entries found) 2023-09-22T13:17:59.047 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.066+0000 7fc46c1d5500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:59.047 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.066+0000 7fc46c1d5500 1 NOTICE: finished shard 5 (0 entries found) 2023-09-22T13:17:59.047 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.066+0000 7fc46c1d5500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:59.047 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.067+0000 7fc46c1d5500 1 NOTICE: finished shard 6 (0 entries found) 2023-09-22T13:17:59.047 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.067+0000 7fc46c1d5500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:59.047 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.067+0000 7fc46c1d5500 1 NOTICE: finished shard 7 (0 entries found) 2023-09-22T13:17:59.047 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.067+0000 7fc46c1d5500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:59.047 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.067+0000 7fc46c1d5500 1 NOTICE: finished shard 8 (0 entries found) 2023-09-22T13:17:59.048 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.068+0000 7fc46c1d5500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:59.048 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.068+0000 7fc46c1d5500 1 NOTICE: finished shard 9 (0 entries found) 2023-09-22T13:17:59.048 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.068+0000 7fc46c1d5500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:59.048 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.068+0000 7fc46c1d5500 1 NOTICE: finished shard 10 (0 entries found) 2023-09-22T13:17:59.048 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.068+0000 7fc46c1d5500 1 NOTICE: finished all shards (0 entries found) 2023-09-22T13:17:59.048 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.069+0000 7fc46c1d5500 20 remove_watcher() i=0 2023-09-22T13:17:59.048 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.069+0000 7fc46c1d5500 2 removed watcher, disabling cache 2023-09-22T13:17:59.048 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.070+0000 7fc46c1d5500 20 remove_watcher() i=1 2023-09-22T13:17:59.048 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.071+0000 7fc46c1d5500 20 remove_watcher() i=2 2023-09-22T13:17:59.048 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.072+0000 7fc46c1d5500 20 remove_watcher() i=3 2023-09-22T13:17:59.048 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.073+0000 7fc46c1d5500 20 remove_watcher() i=4 2023-09-22T13:17:59.048 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.073+0000 7fc46c1d5500 20 remove_watcher() i=5 2023-09-22T13:17:59.049 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.074+0000 7fc46c1d5500 20 remove_watcher() i=6 2023-09-22T13:17:59.049 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.075+0000 7fc46c1d5500 20 remove_watcher() i=7 2023-09-22T13:17:59.049 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:54.075+0000 7fc3a3fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.049 INFO:tasks.workunit.client.0.smithi096.stderr:TEST: verify that bucket check fixes do not affect null version objects 2023-09-22T13:17:59.049 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:59.049 INFO:tasks.workunit.client.0.smithi096.stderr:TEST: bucket check stats are correct in the presence of unlinked entries 2023-09-22T13:17:59.049 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:59.049 INFO:tasks.workunit.client.0.smithi096.stderr:ceph config set client rgw_debug_inject_set_olh_err 2 2023-09-22T13:17:59.049 INFO:tasks.workunit.client.0.smithi096.stderr:ceph config set client rgw_debug_inject_olh_cancel_modification_err true 2023-09-22T13:17:59.049 INFO:tasks.workunit.client.0.smithi096.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-09-22T13:17:59.049 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bi list --bucket check-bucket --object c 2023-09-22T13:17:59.050 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:59.050 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:59.050 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.177+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.050 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.179+0000 7f3435235500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.050 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.179+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.050 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.180+0000 7f3435235500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.050 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.180+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.050 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.180+0000 7f3435235500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.050 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.180+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.050 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.181+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.050 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.193+0000 7f3435235500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:59.050 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.193+0000 7f3435235500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:59.051 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.193+0000 7f3435235500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:59.051 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.193+0000 7f3435235500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:59.051 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.193+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.051 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.194+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.051 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.194+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.051 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.194+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:59.051 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.194+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.051 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.194+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.051 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.194+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.051 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.195+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:59.051 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.195+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.051 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.195+0000 7f3435235500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.052 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.195+0000 7f3435235500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:59.052 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.195+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.052 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.195+0000 7f3435235500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.052 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.195+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.052 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.195+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.052 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.195+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.052 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.196+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:59.052 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.196+0000 7f3435235500 20 zonegroup default 2023-09-22T13:17:59.052 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.196+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.052 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.196+0000 7f3435235500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.052 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.196+0000 7f3435235500 10 Cannot find current period zone using local zone 2023-09-22T13:17:59.053 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.196+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.053 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.196+0000 7f3435235500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.053 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.196+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.053 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.196+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.053 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.196+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.053 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.197+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:59.053 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.197+0000 7f3435235500 20 zone default found 2023-09-22T13:17:59.053 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.197+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.053 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.197+0000 7f3435235500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.053 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.197+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.053 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.198+0000 7f3435235500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.053 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.198+0000 7f3435235500 20 started sync module instance, tier type = 2023-09-22T13:17:59.054 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.198+0000 7f3435235500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:59.054 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.205+0000 7f3435235500 20 add_watcher() i=0 2023-09-22T13:17:59.054 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.205+0000 7f3435235500 20 add_watcher() i=1 2023-09-22T13:17:59.054 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.205+0000 7f3435235500 20 add_watcher() i=2 2023-09-22T13:17:59.054 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.205+0000 7f3435235500 20 add_watcher() i=3 2023-09-22T13:17:59.054 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.205+0000 7f3435235500 20 add_watcher() i=4 2023-09-22T13:17:59.054 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.205+0000 7f3435235500 20 add_watcher() i=5 2023-09-22T13:17:59.054 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.205+0000 7f3435235500 20 add_watcher() i=6 2023-09-22T13:17:59.054 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.206+0000 7f3435235500 20 add_watcher() i=7 2023-09-22T13:17:59.054 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.206+0000 7f3435235500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:59.055 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.206+0000 7f3420f18700 0 -- 172.21.15.96:0/3391173878 >> [v2:172.21.15.96:6816/140810,v1:172.21.15.96:6817/140810] conn(0x562c8d7452a0 msgr2=0x562c8d747690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-09-22T13:17:59.055 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.208+0000 7f33f1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.055 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.208+0000 7f3435235500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:59.055 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.208+0000 7f3435235500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:59.055 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.212+0000 7f3420f18700 0 -- 172.21.15.96:0/3859717637 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562c8d764630 msgr2=0x562c8d788a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-09-22T13:17:59.055 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.215+0000 7f3435235500 5 note: GC not initialized 2023-09-22T13:17:59.055 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.216+0000 7f3368ff9700 20 reqs_thread_entry: start 2023-09-22T13:17:59.055 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.216+0000 7f335e7fc700 20 reqs_thread_entry: start 2023-09-22T13:17:59.055 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.216+0000 7f3435235500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:59.055 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.216+0000 7f3435235500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:59.055 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.216+0000 7f335cff9700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:26 2023 2023-09-22T13:17:59.055 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:59.056 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.216+0000 7f3343fff700 20 reqs_thread_entry: start 2023-09-22T13:17:59.056 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.216+0000 7f3435235500 20 get_system_obj_state: rctx=0x7fffb0e1fbc0 obj=default.rgw.meta:root:check-bucket state=0x562c8d787a30 s->prefetch_data=0 2023-09-22T13:17:59.056 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.217+0000 7f3435235500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.056 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.217+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.056 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.217+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:59.056 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.217+0000 7f3435235500 20 get_system_obj_state: rctx=0x7fffb0e1fbc0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562c8d7614b0 s->prefetch_data=0 2023-09-22T13:17:59.056 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.217+0000 7f3435235500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.056 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.217+0000 7f3435235500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.056 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.217+0000 7f3435235500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.056 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.217+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.056 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.218+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.056 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.218+0000 7f3435235500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.057 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.218+0000 7f3435235500 20 get_system_obj_state: rctx=0x7fffb0e1f990 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562c8d7614b0 s->prefetch_data=0 2023-09-22T13:17:59.057 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.218+0000 7f3435235500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.057 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.218+0000 7f3435235500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.057 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.218+0000 7f3435235500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.057 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.218+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.057 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.218+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.057 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.218+0000 7f3435235500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.057 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.218+0000 7f3435235500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:59.057 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.219+0000 7f3435235500 20 get_system_obj_state: rctx=0x7fffb0e1f990 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562c8d7614b0 s->prefetch_data=0 2023-09-22T13:17:59.057 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.219+0000 7f3435235500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.057 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.219+0000 7f3435235500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.057 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.219+0000 7f3435235500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.058 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.219+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.058 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.219+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.058 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.220+0000 7f3435235500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.058 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.220+0000 7f3435235500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:59.058 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.220+0000 7f3435235500 20 get_system_obj_state: rctx=0x7fffb0e1f990 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562c8d7614b0 s->prefetch_data=0 2023-09-22T13:17:59.058 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.221+0000 7f3435235500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.058 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.221+0000 7f3435235500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.058 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.221+0000 7f3435235500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.058 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.221+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.058 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.221+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.058 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.221+0000 7f3435235500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.059 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.221+0000 7f3435235500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:59.059 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.222+0000 7f3435235500 20 get_system_obj_state: rctx=0x7fffb0e1f990 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562c8d7614b0 s->prefetch_data=0 2023-09-22T13:17:59.059 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.222+0000 7f3435235500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.059 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.222+0000 7f3435235500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.059 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.222+0000 7f3435235500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.059 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.222+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.059 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.223+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.059 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.223+0000 7f3435235500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.059 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.223+0000 7f3435235500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:59.059 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.223+0000 7f3435235500 20 get_system_obj_state: rctx=0x7fffb0e1f990 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562c8d7614b0 s->prefetch_data=0 2023-09-22T13:17:59.059 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.223+0000 7f3435235500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.059 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.223+0000 7f3435235500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.060 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.223+0000 7f3435235500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.060 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.223+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.060 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.224+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.060 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.224+0000 7f3435235500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.060 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.224+0000 7f3435235500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:59.060 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.224+0000 7f3435235500 20 get_system_obj_state: rctx=0x7fffb0e1f990 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562c8d7614b0 s->prefetch_data=0 2023-09-22T13:17:59.060 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.225+0000 7f3435235500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.060 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.225+0000 7f3435235500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.060 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.225+0000 7f3435235500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.060 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.225+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.060 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.225+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.060 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.225+0000 7f3435235500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.061 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.225+0000 7f3435235500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:59.061 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.226+0000 7f3435235500 20 get_system_obj_state: rctx=0x7fffb0e1f990 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562c8d7614b0 s->prefetch_data=0 2023-09-22T13:17:59.061 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.226+0000 7f3435235500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.061 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.226+0000 7f3435235500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.061 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.226+0000 7f3435235500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.061 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.226+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.061 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.226+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.061 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.226+0000 7f3435235500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.061 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.226+0000 7f3435235500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:59.061 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.227+0000 7f3435235500 20 get_system_obj_state: rctx=0x7fffb0e1f990 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562c8d7614b0 s->prefetch_data=0 2023-09-22T13:17:59.061 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.227+0000 7f3435235500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.062 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.227+0000 7f3435235500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.062 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.227+0000 7f3435235500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.062 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.227+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.062 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.227+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.062 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.228+0000 7f3435235500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.062 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.228+0000 7f3435235500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:59.062 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.228+0000 7f3435235500 20 get_system_obj_state: rctx=0x7fffb0e1f990 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562c8d7614b0 s->prefetch_data=0 2023-09-22T13:17:59.062 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.228+0000 7f3435235500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.186 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.228+0000 7f3435235500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.186 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.228+0000 7f3435235500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.187 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.228+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.187 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.229+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.187 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.229+0000 7f3435235500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.187 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.229+0000 7f3435235500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:59.187 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.229+0000 7f3435235500 20 get_system_obj_state: rctx=0x7fffb0e1f990 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562c8d7614b0 s->prefetch_data=0 2023-09-22T13:17:59.187 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.230+0000 7f3435235500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.187 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.230+0000 7f3435235500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.187 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.230+0000 7f3435235500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.187 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.230+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.187 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.230+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.188 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.230+0000 7f3435235500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.188 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.230+0000 7f3435235500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:59.188 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.230+0000 7f3435235500 20 get_system_obj_state: rctx=0x7fffb0e1f990 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562c8d7614b0 s->prefetch_data=0 2023-09-22T13:17:59.188 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.231+0000 7f3435235500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.188 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.231+0000 7f3435235500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.188 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.231+0000 7f3435235500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.188 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.231+0000 7f3435235500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.188 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.231+0000 7f3435235500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.188 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.231+0000 7f3435235500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.188 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.231+0000 7f3435235500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:59.188 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.233+0000 7f3435235500 20 remove_watcher() i=0 2023-09-22T13:17:59.188 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.233+0000 7f3435235500 2 removed watcher, disabling cache 2023-09-22T13:17:59.189 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.234+0000 7f3435235500 20 remove_watcher() i=1 2023-09-22T13:17:59.189 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.235+0000 7f3435235500 20 remove_watcher() i=2 2023-09-22T13:17:59.189 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.236+0000 7f3435235500 20 remove_watcher() i=3 2023-09-22T13:17:59.189 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.237+0000 7f3435235500 20 remove_watcher() i=4 2023-09-22T13:17:59.189 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.237+0000 7f3435235500 20 remove_watcher() i=5 2023-09-22T13:17:59.189 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.238+0000 7f3435235500 20 remove_watcher() i=6 2023-09-22T13:17:59.189 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.239+0000 7f3435235500 20 remove_watcher() i=7 2023-09-22T13:17:59.189 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.240+0000 7f33f1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.189 INFO:tasks.workunit.client.0.smithi096.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-09-22T13:17:59.189 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bi list --bucket check-bucket --object d 2023-09-22T13:17:59.189 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:59.189 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:59.190 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.330+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.190 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.332+0000 7f47c8112500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.190 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.332+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.190 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.333+0000 7f47c8112500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.190 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.333+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.190 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.333+0000 7f47c8112500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.190 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.333+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.190 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.334+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.190 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.344+0000 7f47c8112500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:59.190 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.344+0000 7f47c8112500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:59.190 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.344+0000 7f47c8112500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:59.190 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.344+0000 7f47c8112500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:59.191 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.344+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.191 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.344+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.191 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.344+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.191 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.345+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:59.191 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.345+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.191 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.345+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.191 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.345+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.191 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.345+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:59.191 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.345+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.191 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.345+0000 7f47c8112500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.191 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.345+0000 7f47c8112500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:59.192 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.345+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.192 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.346+0000 7f47c8112500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.192 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.346+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.192 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.346+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.192 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.346+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.192 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.346+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:59.192 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.346+0000 7f47c8112500 20 zonegroup default 2023-09-22T13:17:59.192 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.346+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.192 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.346+0000 7f47c8112500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.192 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.346+0000 7f47c8112500 10 Cannot find current period zone using local zone 2023-09-22T13:17:59.192 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.346+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.192 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.347+0000 7f47c8112500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.192 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.347+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.193 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.347+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.193 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.347+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.193 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.347+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:59.193 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.347+0000 7f47c8112500 20 zone default found 2023-09-22T13:17:59.193 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.347+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.193 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.348+0000 7f47c8112500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.193 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.348+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.193 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.348+0000 7f47c8112500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.193 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.348+0000 7f47c8112500 20 started sync module instance, tier type = 2023-09-22T13:17:59.193 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.348+0000 7f47c8112500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:59.193 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.354+0000 7f47c8112500 20 add_watcher() i=0 2023-09-22T13:17:59.194 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.354+0000 7f47c8112500 20 add_watcher() i=1 2023-09-22T13:17:59.194 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.354+0000 7f47c8112500 20 add_watcher() i=2 2023-09-22T13:17:59.194 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.354+0000 7f47c8112500 20 add_watcher() i=3 2023-09-22T13:17:59.194 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.354+0000 7f47c8112500 20 add_watcher() i=4 2023-09-22T13:17:59.194 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.354+0000 7f47c8112500 20 add_watcher() i=5 2023-09-22T13:17:59.194 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.354+0000 7f47c8112500 20 add_watcher() i=6 2023-09-22T13:17:59.194 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.355+0000 7f47c8112500 20 add_watcher() i=7 2023-09-22T13:17:59.194 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.355+0000 7f47c8112500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:59.194 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.355+0000 7f47b3df5700 0 -- 172.21.15.96:0/1424878369 >> [v2:172.21.15.96:6816/140810,v1:172.21.15.96:6817/140810] conn(0x5648174d2600 msgr2=0x5648174d49f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:59.194 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.357+0000 7f46fffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.194 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.357+0000 7f47c8112500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:59.195 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.357+0000 7f47c8112500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:59.195 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.361+0000 7f47b3df5700 0 -- 172.21.15.96:0/1698496527 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x564817511af0 msgr2=0x5648175121b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:59.195 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.363+0000 7f47c8112500 5 note: GC not initialized 2023-09-22T13:17:59.195 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.364+0000 7f46f6ffd700 20 reqs_thread_entry: start 2023-09-22T13:17:59.195 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.364+0000 7f46f4ff9700 20 reqs_thread_entry: start 2023-09-22T13:17:59.195 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.364+0000 7f47c8112500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:59.195 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.365+0000 7f47c8112500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:59.195 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.365+0000 7f46e2ffd700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:26 2023 2023-09-22T13:17:59.195 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:59.195 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.365+0000 7f46e27fc700 20 reqs_thread_entry: start 2023-09-22T13:17:59.195 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.365+0000 7f47c8112500 20 get_system_obj_state: rctx=0x7ffd225aad80 obj=default.rgw.meta:root:check-bucket state=0x564817365810 s->prefetch_data=0 2023-09-22T13:17:59.195 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.365+0000 7f47c8112500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.195 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.365+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.196 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.365+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:59.196 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.365+0000 7f47c8112500 20 get_system_obj_state: rctx=0x7ffd225aad80 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x56481731f910 s->prefetch_data=0 2023-09-22T13:17:59.196 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.366+0000 7f47c8112500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.196 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.366+0000 7f47c8112500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.196 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.366+0000 7f47c8112500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.196 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.366+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.196 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.366+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.196 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.366+0000 7f47c8112500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.196 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.366+0000 7f47c8112500 20 get_system_obj_state: rctx=0x7ffd225aab50 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x56481731f910 s->prefetch_data=0 2023-09-22T13:17:59.196 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.366+0000 7f47c8112500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.196 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.366+0000 7f47c8112500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.196 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.366+0000 7f47c8112500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.197 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.366+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.197 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.367+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.197 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.367+0000 7f47c8112500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.197 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.367+0000 7f47c8112500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:59.197 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.367+0000 7f47c8112500 20 get_system_obj_state: rctx=0x7ffd225aab50 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x56481731f910 s->prefetch_data=0 2023-09-22T13:17:59.197 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.367+0000 7f47c8112500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.197 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.367+0000 7f47c8112500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.197 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.367+0000 7f47c8112500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.197 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.367+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.197 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.368+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.197 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.368+0000 7f47c8112500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.197 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.368+0000 7f47c8112500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:59.197 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.368+0000 7f47c8112500 20 get_system_obj_state: rctx=0x7ffd225aab50 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x56481731f910 s->prefetch_data=0 2023-09-22T13:17:59.198 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.369+0000 7f47c8112500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.198 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.369+0000 7f47c8112500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.198 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.369+0000 7f47c8112500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.198 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.369+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.198 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.369+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.198 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.369+0000 7f47c8112500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.198 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.369+0000 7f47c8112500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:59.198 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.370+0000 7f47c8112500 20 get_system_obj_state: rctx=0x7ffd225aab50 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x56481731f910 s->prefetch_data=0 2023-09-22T13:17:59.199 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.370+0000 7f47c8112500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.199 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.370+0000 7f47c8112500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.199 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.370+0000 7f47c8112500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.199 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.370+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.199 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.370+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.199 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.370+0000 7f47c8112500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.199 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.370+0000 7f47c8112500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:59.199 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.371+0000 7f47c8112500 20 get_system_obj_state: rctx=0x7ffd225aab50 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x56481731f910 s->prefetch_data=0 2023-09-22T13:17:59.199 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.371+0000 7f47c8112500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.199 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.371+0000 7f47c8112500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.199 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.371+0000 7f47c8112500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.199 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.371+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.199 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.371+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.200 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.371+0000 7f47c8112500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.200 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.371+0000 7f47c8112500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:59.200 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.372+0000 7f47c8112500 20 get_system_obj_state: rctx=0x7ffd225aab50 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x56481731f910 s->prefetch_data=0 2023-09-22T13:17:59.200 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.372+0000 7f47c8112500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.200 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.372+0000 7f47c8112500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.200 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.372+0000 7f47c8112500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.200 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.372+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.200 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.372+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.200 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.372+0000 7f47c8112500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.200 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.372+0000 7f47c8112500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:59.200 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.373+0000 7f47c8112500 20 get_system_obj_state: rctx=0x7ffd225aab50 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x56481731f910 s->prefetch_data=0 2023-09-22T13:17:59.200 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.374+0000 7f47c8112500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.201 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.374+0000 7f47c8112500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.201 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.374+0000 7f47c8112500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.201 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.374+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.201 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.374+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.201 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.374+0000 7f47c8112500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.201 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.374+0000 7f47c8112500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:59.201 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.374+0000 7f47c8112500 20 get_system_obj_state: rctx=0x7ffd225aab50 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x56481731f910 s->prefetch_data=0 2023-09-22T13:17:59.201 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.375+0000 7f47c8112500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.201 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.375+0000 7f47c8112500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.201 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.375+0000 7f47c8112500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.201 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.375+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.201 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.375+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.201 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.375+0000 7f47c8112500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.202 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.375+0000 7f47c8112500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:59.202 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.375+0000 7f47c8112500 20 get_system_obj_state: rctx=0x7ffd225aab50 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x56481731f910 s->prefetch_data=0 2023-09-22T13:17:59.202 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.376+0000 7f47c8112500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.202 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.376+0000 7f47c8112500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.202 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.376+0000 7f47c8112500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.202 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.376+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.202 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.376+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.202 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.376+0000 7f47c8112500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.202 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.376+0000 7f47c8112500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:59.202 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.377+0000 7f47c8112500 20 get_system_obj_state: rctx=0x7ffd225aab50 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x56481731f910 s->prefetch_data=0 2023-09-22T13:17:59.202 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.377+0000 7f47c8112500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.202 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.377+0000 7f47c8112500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.203 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.377+0000 7f47c8112500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.203 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.377+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.203 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.377+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.203 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.377+0000 7f47c8112500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.203 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.377+0000 7f47c8112500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:59.203 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.378+0000 7f47c8112500 20 get_system_obj_state: rctx=0x7ffd225aab50 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x56481731f910 s->prefetch_data=0 2023-09-22T13:17:59.203 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.378+0000 7f47c8112500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.203 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.378+0000 7f47c8112500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.203 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.378+0000 7f47c8112500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.203 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.378+0000 7f47c8112500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.203 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.378+0000 7f47c8112500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.203 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.378+0000 7f47c8112500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.204 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.378+0000 7f47c8112500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:59.204 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.380+0000 7f47c8112500 20 remove_watcher() i=0 2023-09-22T13:17:59.204 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.380+0000 7f47c8112500 2 removed watcher, disabling cache 2023-09-22T13:17:59.204 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.380+0000 7f47c8112500 20 remove_watcher() i=1 2023-09-22T13:17:59.204 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.382+0000 7f47c8112500 20 remove_watcher() i=2 2023-09-22T13:17:59.204 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.382+0000 7f47c8112500 20 remove_watcher() i=3 2023-09-22T13:17:59.204 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.383+0000 7f47c8112500 20 remove_watcher() i=4 2023-09-22T13:17:59.204 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.384+0000 7f47c8112500 20 remove_watcher() i=5 2023-09-22T13:17:59.204 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.385+0000 7f47c8112500 20 remove_watcher() i=6 2023-09-22T13:17:59.204 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.386+0000 7f47c8112500 20 remove_watcher() i=7 2023-09-22T13:17:59.204 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.386+0000 7f46fffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.204 INFO:tasks.workunit.client.0.smithi096.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-09-22T13:17:59.205 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bi list --bucket check-bucket --object e 2023-09-22T13:17:59.205 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:59.205 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:59.205 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.477+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.205 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.479+0000 7f1d2278b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.205 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.479+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.205 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.479+0000 7f1d2278b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.205 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.479+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.205 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.480+0000 7f1d2278b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.205 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.480+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.205 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.481+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.205 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.490+0000 7f1d2278b500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:59.206 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.490+0000 7f1d2278b500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:59.206 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.490+0000 7f1d2278b500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:59.206 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.490+0000 7f1d2278b500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:59.206 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.490+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.206 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.491+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.206 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.491+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.206 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.491+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:59.206 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.491+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.206 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.491+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.206 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.491+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.206 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.491+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:59.206 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.491+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.206 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.492+0000 7f1d2278b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.207 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.492+0000 7f1d2278b500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:59.207 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.492+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.207 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.492+0000 7f1d2278b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.207 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.492+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.207 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.492+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.207 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.492+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.207 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.492+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:59.207 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.492+0000 7f1d2278b500 20 zonegroup default 2023-09-22T13:17:59.207 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.492+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.207 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.493+0000 7f1d2278b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.207 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.493+0000 7f1d2278b500 10 Cannot find current period zone using local zone 2023-09-22T13:17:59.207 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.493+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.208 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.493+0000 7f1d2278b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.208 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.493+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.208 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.493+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.208 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.493+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.208 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.493+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:59.208 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.493+0000 7f1d2278b500 20 zone default found 2023-09-22T13:17:59.208 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.493+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.208 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.494+0000 7f1d2278b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.208 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.494+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.208 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.494+0000 7f1d2278b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.208 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.494+0000 7f1d2278b500 20 started sync module instance, tier type = 2023-09-22T13:17:59.208 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.494+0000 7f1d2278b500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:59.209 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.500+0000 7f1d0dc6d700 0 -- 172.21.15.96:0/3895674914 >> [v2:172.21.15.96:6807/140797,v1:172.21.15.96:6809/140797] conn(0x7f1d00031950 msgr2=0x7f1d000303d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:59.209 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.501+0000 7f1d2278b500 20 add_watcher() i=0 2023-09-22T13:17:59.209 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.501+0000 7f1d2278b500 20 add_watcher() i=1 2023-09-22T13:17:59.209 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.501+0000 7f1d2278b500 20 add_watcher() i=2 2023-09-22T13:17:59.209 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.501+0000 7f1d2278b500 20 add_watcher() i=3 2023-09-22T13:17:59.209 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.501+0000 7f1d2278b500 20 add_watcher() i=4 2023-09-22T13:17:59.209 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.501+0000 7f1d2278b500 20 add_watcher() i=5 2023-09-22T13:17:59.209 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.501+0000 7f1d2278b500 20 add_watcher() i=6 2023-09-22T13:17:59.209 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.501+0000 7f1d2278b500 20 add_watcher() i=7 2023-09-22T13:17:59.209 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.501+0000 7f1d2278b500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:59.209 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.503+0000 7f1cdf7de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.209 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.503+0000 7f1d2278b500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:59.209 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.503+0000 7f1d2278b500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:59.210 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.507+0000 7f1d0e46e700 0 -- 172.21.15.96:0/688594988 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55f1a96da8c0 msgr2=0x55f1a96e8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:59.210 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.509+0000 7f1d2278b500 5 note: GC not initialized 2023-09-22T13:17:59.210 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.510+0000 7f1c4e7fc700 20 reqs_thread_entry: start 2023-09-22T13:17:59.210 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.510+0000 7f1c3ffff700 20 reqs_thread_entry: start 2023-09-22T13:17:59.210 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.511+0000 7f1d2278b500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:59.210 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.511+0000 7f1d2278b500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:59.210 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.511+0000 7f1c3e7fc700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:26 2023 2023-09-22T13:17:59.210 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:59.210 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.511+0000 7f1c3dffb700 20 reqs_thread_entry: start 2023-09-22T13:17:59.210 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.511+0000 7f1d2278b500 20 get_system_obj_state: rctx=0x7ffe9207d070 obj=default.rgw.meta:root:check-bucket state=0x55f1a96e7c00 s->prefetch_data=0 2023-09-22T13:17:59.210 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.511+0000 7f1d2278b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.338 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.511+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.511+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:59.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.511+0000 7f1d2278b500 20 get_system_obj_state: rctx=0x7ffe9207d070 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55f1a9666400 s->prefetch_data=0 2023-09-22T13:17:59.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.512+0000 7f1d2278b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.512+0000 7f1d2278b500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.512+0000 7f1d2278b500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.512+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.512+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.512+0000 7f1d2278b500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.512+0000 7f1d2278b500 20 get_system_obj_state: rctx=0x7ffe9207ce40 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55f1a9666400 s->prefetch_data=0 2023-09-22T13:17:59.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.512+0000 7f1d2278b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.512+0000 7f1d2278b500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.512+0000 7f1d2278b500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.512+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.513+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.513+0000 7f1d2278b500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.513+0000 7f1d2278b500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:59.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.513+0000 7f1d2278b500 20 get_system_obj_state: rctx=0x7ffe9207ce40 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55f1a9666400 s->prefetch_data=0 2023-09-22T13:17:59.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.514+0000 7f1d2278b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.514+0000 7f1d2278b500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.514+0000 7f1d2278b500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.514+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.514+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.514+0000 7f1d2278b500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.514+0000 7f1d2278b500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:59.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.515+0000 7f1d2278b500 20 get_system_obj_state: rctx=0x7ffe9207ce40 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55f1a9666400 s->prefetch_data=0 2023-09-22T13:17:59.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.515+0000 7f1d2278b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.515+0000 7f1d2278b500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.515+0000 7f1d2278b500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.515+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.515+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.515+0000 7f1d2278b500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.341 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.515+0000 7f1d2278b500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:59.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.516+0000 7f1d2278b500 20 get_system_obj_state: rctx=0x7ffe9207ce40 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55f1a9666400 s->prefetch_data=0 2023-09-22T13:17:59.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.516+0000 7f1d2278b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.516+0000 7f1d2278b500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.516+0000 7f1d2278b500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.516+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.516+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.516+0000 7f1d2278b500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.516+0000 7f1d2278b500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:59.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.517+0000 7f1d2278b500 20 get_system_obj_state: rctx=0x7ffe9207ce40 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55f1a9666400 s->prefetch_data=0 2023-09-22T13:17:59.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.517+0000 7f1d2278b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.342 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.517+0000 7f1d2278b500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.517+0000 7f1d2278b500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.517+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.518+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.518+0000 7f1d2278b500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.518+0000 7f1d2278b500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:59.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.518+0000 7f1d2278b500 20 get_system_obj_state: rctx=0x7ffe9207ce40 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55f1a9666400 s->prefetch_data=0 2023-09-22T13:17:59.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.518+0000 7f1d2278b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.518+0000 7f1d2278b500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.518+0000 7f1d2278b500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.518+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.519+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.343 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.519+0000 7f1d2278b500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.344 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.519+0000 7f1d2278b500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:59.344 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.519+0000 7f1d2278b500 20 get_system_obj_state: rctx=0x7ffe9207ce40 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55f1a9666400 s->prefetch_data=0 2023-09-22T13:17:59.344 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.520+0000 7f1d2278b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.344 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.520+0000 7f1d2278b500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.344 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.520+0000 7f1d2278b500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.344 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.520+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.344 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.520+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.344 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.520+0000 7f1d2278b500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.344 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.520+0000 7f1d2278b500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:59.344 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.521+0000 7f1d2278b500 20 get_system_obj_state: rctx=0x7ffe9207ce40 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55f1a9666400 s->prefetch_data=0 2023-09-22T13:17:59.344 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.521+0000 7f1d2278b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.345 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.521+0000 7f1d2278b500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.345 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.521+0000 7f1d2278b500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.345 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.521+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.345 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.521+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.345 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.521+0000 7f1d2278b500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.345 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.521+0000 7f1d2278b500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:59.345 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.522+0000 7f1d2278b500 20 get_system_obj_state: rctx=0x7ffe9207ce40 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55f1a9666400 s->prefetch_data=0 2023-09-22T13:17:59.345 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.522+0000 7f1d2278b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.345 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.522+0000 7f1d2278b500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.345 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.522+0000 7f1d2278b500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.345 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.522+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.345 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.522+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.346 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.522+0000 7f1d2278b500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.346 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.522+0000 7f1d2278b500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:59.346 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.523+0000 7f1d2278b500 20 get_system_obj_state: rctx=0x7ffe9207ce40 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55f1a9666400 s->prefetch_data=0 2023-09-22T13:17:59.346 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.523+0000 7f1d2278b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.346 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.523+0000 7f1d2278b500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.346 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.523+0000 7f1d2278b500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.346 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.523+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.346 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.524+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.346 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.524+0000 7f1d2278b500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.346 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.524+0000 7f1d2278b500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:59.346 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.524+0000 7f1d2278b500 20 get_system_obj_state: rctx=0x7ffe9207ce40 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x55f1a9666400 s->prefetch_data=0 2023-09-22T13:17:59.347 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.524+0000 7f1d2278b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.347 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.524+0000 7f1d2278b500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.347 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.524+0000 7f1d2278b500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.347 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.524+0000 7f1d2278b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.347 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.525+0000 7f1d2278b500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.347 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.525+0000 7f1d2278b500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.347 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.525+0000 7f1d2278b500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:59.347 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.526+0000 7f1d2278b500 20 remove_watcher() i=0 2023-09-22T13:17:59.347 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.526+0000 7f1d2278b500 2 removed watcher, disabling cache 2023-09-22T13:17:59.347 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.527+0000 7f1d2278b500 20 remove_watcher() i=1 2023-09-22T13:17:59.347 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.528+0000 7f1d2278b500 20 remove_watcher() i=2 2023-09-22T13:17:59.348 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.529+0000 7f1d2278b500 20 remove_watcher() i=3 2023-09-22T13:17:59.348 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.530+0000 7f1d2278b500 20 remove_watcher() i=4 2023-09-22T13:17:59.348 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.531+0000 7f1d2278b500 20 remove_watcher() i=5 2023-09-22T13:17:59.348 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.532+0000 7f1d2278b500 20 remove_watcher() i=6 2023-09-22T13:17:59.348 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.532+0000 7f1d2278b500 20 remove_watcher() i=7 2023-09-22T13:17:59.348 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.533+0000 7f1cdf7de700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.348 INFO:tasks.workunit.client.0.smithi096.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-09-22T13:17:59.348 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bi list --bucket check-bucket --object f 2023-09-22T13:17:59.348 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:59.348 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:59.348 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.621+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.348 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.623+0000 7f166e1ee500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.349 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.623+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.349 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.623+0000 7f166e1ee500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.349 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.623+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.349 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.623+0000 7f166e1ee500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.349 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.623+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.349 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.624+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.349 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.634+0000 7f166e1ee500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:59.349 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.634+0000 7f166e1ee500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:59.349 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.634+0000 7f166e1ee500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:59.349 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.634+0000 7f166e1ee500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:59.349 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.634+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.349 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.634+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.350 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.634+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.350 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.634+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:59.350 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.635+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.350 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.635+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.350 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.635+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.350 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.635+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:59.350 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.635+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.350 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.636+0000 7f166e1ee500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.350 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.636+0000 7f166e1ee500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:59.350 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.636+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.350 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.636+0000 7f166e1ee500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.351 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.636+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.351 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.636+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.351 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.636+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.351 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.636+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:59.351 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.636+0000 7f166e1ee500 20 zonegroup default 2023-09-22T13:17:59.351 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.636+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.351 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.637+0000 7f166e1ee500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.351 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.637+0000 7f166e1ee500 10 Cannot find current period zone using local zone 2023-09-22T13:17:59.351 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.637+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.351 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.637+0000 7f166e1ee500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.351 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.637+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.351 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.637+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.352 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.637+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.352 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.637+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:59.352 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.637+0000 7f166e1ee500 20 zone default found 2023-09-22T13:17:59.352 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.637+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.352 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.638+0000 7f166e1ee500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.352 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.638+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.352 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.638+0000 7f166e1ee500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.352 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.638+0000 7f166e1ee500 20 started sync module instance, tier type = 2023-09-22T13:17:59.352 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.638+0000 7f166e1ee500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:59.352 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.645+0000 7f166e1ee500 20 add_watcher() i=0 2023-09-22T13:17:59.352 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.645+0000 7f166e1ee500 20 add_watcher() i=1 2023-09-22T13:17:59.352 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.645+0000 7f166e1ee500 20 add_watcher() i=2 2023-09-22T13:17:59.353 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.645+0000 7f166e1ee500 20 add_watcher() i=3 2023-09-22T13:17:59.353 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.645+0000 7f166e1ee500 20 add_watcher() i=4 2023-09-22T13:17:59.353 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.645+0000 7f166e1ee500 20 add_watcher() i=5 2023-09-22T13:17:59.353 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.645+0000 7f166e1ee500 20 add_watcher() i=6 2023-09-22T13:17:59.353 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.645+0000 7f166e1ee500 20 add_watcher() i=7 2023-09-22T13:17:59.353 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.645+0000 7f166e1ee500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:59.353 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.645+0000 7f1659ed1700 0 -- 172.21.15.96:0/4052663836 >> [v2:172.21.15.96:6816/140810,v1:172.21.15.96:6817/140810] conn(0x562ffdf29230 msgr2=0x562ffdf2b620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-09-22T13:17:59.353 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.647+0000 7f162a7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.353 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.647+0000 7f166e1ee500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:59.353 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.647+0000 7f166e1ee500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:59.353 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.651+0000 7f16596d0700 0 -- 172.21.15.96:0/1294187537 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ffdf71160 msgr2=0x562ffdf7b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:59.353 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.654+0000 7f166e1ee500 5 note: GC not initialized 2023-09-22T13:17:59.354 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.655+0000 7f15997fa700 20 reqs_thread_entry: start 2023-09-22T13:17:59.354 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.655+0000 7f158affd700 20 reqs_thread_entry: start 2023-09-22T13:17:59.354 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.655+0000 7f166e1ee500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:59.354 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.655+0000 7f166e1ee500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:59.354 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.655+0000 7f15897fa700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:26 2023 2023-09-22T13:17:59.354 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:59.354 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.655+0000 7f1588ff9700 20 reqs_thread_entry: start 2023-09-22T13:17:59.354 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.655+0000 7f166e1ee500 20 get_system_obj_state: rctx=0x7ffedb18b1e0 obj=default.rgw.meta:root:check-bucket state=0x562ffdee0640 s->prefetch_data=0 2023-09-22T13:17:59.354 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.655+0000 7f166e1ee500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.354 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.655+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.354 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.656+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:59.355 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.656+0000 7f166e1ee500 20 get_system_obj_state: rctx=0x7ffedb18b1e0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562ffddbd810 s->prefetch_data=0 2023-09-22T13:17:59.355 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.656+0000 7f166e1ee500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.355 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.656+0000 7f166e1ee500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.355 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.656+0000 7f166e1ee500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.355 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.656+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.355 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.657+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.355 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.657+0000 7f166e1ee500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.355 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.657+0000 7f166e1ee500 20 get_system_obj_state: rctx=0x7ffedb18afb0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562ffddbd810 s->prefetch_data=0 2023-09-22T13:17:59.355 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.657+0000 7f166e1ee500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.355 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.657+0000 7f166e1ee500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.355 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.657+0000 7f166e1ee500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.356 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.657+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.356 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.657+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.356 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.657+0000 7f166e1ee500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.356 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.657+0000 7f166e1ee500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:59.356 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.658+0000 7f166e1ee500 20 get_system_obj_state: rctx=0x7ffedb18afb0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562ffddbd810 s->prefetch_data=0 2023-09-22T13:17:59.356 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.658+0000 7f166e1ee500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.356 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.658+0000 7f166e1ee500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.356 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.658+0000 7f166e1ee500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.356 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.658+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.356 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.658+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.356 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.658+0000 7f166e1ee500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.659+0000 7f166e1ee500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:59.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.659+0000 7f166e1ee500 20 get_system_obj_state: rctx=0x7ffedb18afb0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562ffddbd810 s->prefetch_data=0 2023-09-22T13:17:59.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.660+0000 7f166e1ee500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.660+0000 7f166e1ee500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.660+0000 7f166e1ee500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.660+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.660+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.660+0000 7f166e1ee500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.660+0000 7f166e1ee500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:59.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.660+0000 7f166e1ee500 20 get_system_obj_state: rctx=0x7ffedb18afb0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562ffddbd810 s->prefetch_data=0 2023-09-22T13:17:59.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.661+0000 7f166e1ee500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.661+0000 7f166e1ee500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.661+0000 7f166e1ee500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.661+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.661+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.661+0000 7f166e1ee500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.661+0000 7f166e1ee500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:59.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.661+0000 7f166e1ee500 20 get_system_obj_state: rctx=0x7ffedb18afb0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562ffddbd810 s->prefetch_data=0 2023-09-22T13:17:59.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.662+0000 7f166e1ee500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.662+0000 7f166e1ee500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.662+0000 7f166e1ee500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.662+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.662+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.662+0000 7f166e1ee500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.662+0000 7f166e1ee500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:59.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.663+0000 7f166e1ee500 20 get_system_obj_state: rctx=0x7ffedb18afb0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562ffddbd810 s->prefetch_data=0 2023-09-22T13:17:59.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.663+0000 7f166e1ee500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.663+0000 7f166e1ee500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.663+0000 7f166e1ee500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.663+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.663+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.663+0000 7f166e1ee500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.663+0000 7f166e1ee500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:59.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.664+0000 7f166e1ee500 20 get_system_obj_state: rctx=0x7ffedb18afb0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562ffddbd810 s->prefetch_data=0 2023-09-22T13:17:59.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.664+0000 7f166e1ee500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.664+0000 7f166e1ee500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.664+0000 7f166e1ee500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.664+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.664+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.664+0000 7f166e1ee500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.664+0000 7f166e1ee500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:59.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.665+0000 7f166e1ee500 20 get_system_obj_state: rctx=0x7ffedb18afb0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562ffddbd810 s->prefetch_data=0 2023-09-22T13:17:59.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.665+0000 7f166e1ee500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.665+0000 7f166e1ee500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.665+0000 7f166e1ee500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.360 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.665+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.361 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.665+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.361 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.665+0000 7f166e1ee500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.361 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.665+0000 7f166e1ee500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:59.361 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.666+0000 7f166e1ee500 20 get_system_obj_state: rctx=0x7ffedb18afb0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562ffddbd810 s->prefetch_data=0 2023-09-22T13:17:59.361 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.666+0000 7f166e1ee500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.361 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.666+0000 7f166e1ee500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.361 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.666+0000 7f166e1ee500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.361 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.666+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.361 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.667+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.361 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.667+0000 7f166e1ee500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.361 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.667+0000 7f166e1ee500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:59.361 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.667+0000 7f166e1ee500 20 get_system_obj_state: rctx=0x7ffedb18afb0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562ffddbd810 s->prefetch_data=0 2023-09-22T13:17:59.362 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.668+0000 7f166e1ee500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.362 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.668+0000 7f166e1ee500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.362 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.668+0000 7f166e1ee500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.362 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.668+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.362 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.668+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.362 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.668+0000 7f166e1ee500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.362 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.668+0000 7f166e1ee500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:59.362 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.669+0000 7f166e1ee500 20 get_system_obj_state: rctx=0x7ffedb18afb0 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562ffddbd810 s->prefetch_data=0 2023-09-22T13:17:59.362 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.669+0000 7f166e1ee500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.362 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.669+0000 7f166e1ee500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.669+0000 7f166e1ee500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.669+0000 7f166e1ee500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.669+0000 7f166e1ee500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.669+0000 7f166e1ee500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.669+0000 7f166e1ee500 20 bucket index oid: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:59.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.671+0000 7f166e1ee500 20 remove_watcher() i=0 2023-09-22T13:17:59.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.671+0000 7f166e1ee500 2 removed watcher, disabling cache 2023-09-22T13:17:59.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.671+0000 7f166e1ee500 20 remove_watcher() i=1 2023-09-22T13:17:59.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.672+0000 7f166e1ee500 20 remove_watcher() i=2 2023-09-22T13:17:59.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.673+0000 7f166e1ee500 20 remove_watcher() i=3 2023-09-22T13:17:59.363 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.674+0000 7f166e1ee500 20 remove_watcher() i=4 2023-09-22T13:17:59.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.675+0000 7f166e1ee500 20 remove_watcher() i=5 2023-09-22T13:17:59.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.675+0000 7f166e1ee500 20 remove_watcher() i=6 2023-09-22T13:17:59.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.676+0000 7f166e1ee500 20 remove_watcher() i=7 2023-09-22T13:17:59.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:56.677+0000 7f162a7e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.364 INFO:tasks.workunit.client.0.smithi096.stderr:ceph config rm client rgw_debug_inject_set_olh_err 2023-09-22T13:17:59.364 INFO:tasks.workunit.client.0.smithi096.stderr:ceph config rm client rgw_debug_inject_olh_cancel_modification_err 2023-09-22T13:17:59.364 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bucket check --fix --bucket check-bucket 2023-09-22T13:17:59.364 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:59.364 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:59.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.521+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.523+0000 7f88d788c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.523+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.524+0000 7f88d788c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.524+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.524+0000 7f88d788c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.524+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.525+0000 7f88d788c500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.535+0000 7f88d788c500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:59.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.535+0000 7f88d788c500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:59.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.535+0000 7f88d788c500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:59.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.535+0000 7f88d788c500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:59.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.535+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.535+0000 7f88d788c500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.535+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.536+0000 7f88d788c500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:59.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.536+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.536+0000 7f88d788c500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.536+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.536+0000 7f88d788c500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:59.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.536+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.536+0000 7f88d788c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.536+0000 7f88d788c500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:59.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.536+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.537+0000 7f88d788c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.537+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.537+0000 7f88d788c500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.537+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.537+0000 7f88d788c500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:59.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.537+0000 7f88d788c500 20 zonegroup default 2023-09-22T13:17:59.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.537+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.537+0000 7f88d788c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.537+0000 7f88d788c500 10 Cannot find current period zone using local zone 2023-09-22T13:17:59.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.537+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.538+0000 7f88d788c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.538+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.538+0000 7f88d788c500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.538+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.368 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.538+0000 7f88d788c500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:59.368 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.538+0000 7f88d788c500 20 zone default found 2023-09-22T13:17:59.368 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.538+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.368 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.539+0000 7f88d788c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.495 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.539+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.496 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.539+0000 7f88d788c500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.496 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.539+0000 7f88d788c500 20 started sync module instance, tier type = 2023-09-22T13:17:59.496 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.539+0000 7f88d788c500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:59.496 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.544+0000 7f88c2d6e700 0 -- 172.21.15.96:0/2950899122 >> [v2:172.21.15.96:6824/140822,v1:172.21.15.96:6825/140822] conn(0x564205b6eaa0 msgr2=0x564205b70e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-09-22T13:17:59.496 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.546+0000 7f88d788c500 20 add_watcher() i=0 2023-09-22T13:17:59.496 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.546+0000 7f88d788c500 20 add_watcher() i=1 2023-09-22T13:17:59.496 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.546+0000 7f88d788c500 20 add_watcher() i=2 2023-09-22T13:17:59.496 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.546+0000 7f88d788c500 20 add_watcher() i=3 2023-09-22T13:17:59.496 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.546+0000 7f88d788c500 20 add_watcher() i=4 2023-09-22T13:17:59.497 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.546+0000 7f88d788c500 20 add_watcher() i=5 2023-09-22T13:17:59.497 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.546+0000 7f88d788c500 20 add_watcher() i=6 2023-09-22T13:17:59.497 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.547+0000 7f88d788c500 20 add_watcher() i=7 2023-09-22T13:17:59.497 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.547+0000 7f88d788c500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:59.497 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.548+0000 7f8893fe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.497 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.548+0000 7f88d788c500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:59.497 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.548+0000 7f88d788c500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:59.497 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.552+0000 7f88c356f700 0 -- 172.21.15.96:0/806683115 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x564205ba3d60 msgr2=0x564205bc1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:59.497 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.554+0000 7f88d788c500 5 note: GC not initialized 2023-09-22T13:17:59.497 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.555+0000 7f8802ffd700 20 reqs_thread_entry: start 2023-09-22T13:17:59.497 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.555+0000 7f8800ff9700 20 reqs_thread_entry: start 2023-09-22T13:17:59.497 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.555+0000 7f88d788c500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:59.498 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.555+0000 7f88d788c500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:59.498 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.555+0000 7f87f2ffd700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:27 2023 2023-09-22T13:17:59.498 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:59.498 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.555+0000 7f87f27fc700 20 reqs_thread_entry: start 2023-09-22T13:17:59.498 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.555+0000 7f88d788c500 20 get_system_obj_state: rctx=0x7fff318d1828 obj=default.rgw.meta:root:check-bucket state=0x564205ba6220 s->prefetch_data=0 2023-09-22T13:17:59.498 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.555+0000 7f88d788c500 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-09-22T13:17:59.498 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x16 2023-09-22T13:17:59.498 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:59.498 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:59.498 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.498 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 10 cache get: name=default.rgw.meta+root+check-bucket : type miss (requested=0x11, cached=0x16) 2023-09-22T13:17:59.499 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.499 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:59.499 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-09-22T13:17:59.499 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 10 moving default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:59.499 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 20 get_system_obj_state: rctx=0x7fff318d18f8 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x564205ba6220 s->prefetch_data=0 2023-09-22T13:17:59.499 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : miss 2023-09-22T13:17:59.499 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x16 2023-09-22T13:17:59.499 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:59.499 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:59.499 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:59.499 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.499 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.500 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.500 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : type miss (requested=0x13, cached=0x16) 2023-09-22T13:17:59.500 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.556+0000 7f88d788c500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.500 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.557+0000 7f88d788c500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.500 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.557+0000 7f88d788c500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x13 2023-09-22T13:17:59.500 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.557+0000 7f88d788c500 10 moving default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:59.500 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.557+0000 7f88d788c500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:59.500 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.557+0000 7f88d788c500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:59.500 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.557+0000 7f88d788c500 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 2023-09-22T13:17:59.500 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.557+0000 7f88d788c500 20 RGWRados::Bucket::List::list_objects_ordered starting attempt 1 2023-09-22T13:17:59.500 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.557+0000 7f88d788c500 10 RGWRados::cls_bucket_list_ordered: :check-bucket[42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1]) start_after="_multipart_[]", prefix="_multipart_" num_entries=1001, list_versions=1, expansion_factor=1 2023-09-22T13:17:59.501 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.557+0000 7f88d788c500 10 RGWRados::cls_bucket_list_ordered request from each of 11 shard(s) for 112 entries to get 1001 total entries 2023-09-22T13:17:59.501 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.558+0000 7f88d788c500 20 RGWRados::cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-09-22T13:17:59.501 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.558+0000 7f88d788c500 20 RGWRados::cls_bucket_list_ordered: returning, last_entry NOT SET 2023-09-22T13:17:59.501 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.558+0000 7f88d788c500 20 RGWRados::Bucket::List::list_objects_ordered INFO end of outer loop, truncated=0, count=0, attempt=1 2023-09-22T13:17:59.501 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.563+0000 7f88d788c500 20 remove_watcher() i=0 2023-09-22T13:17:59.501 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.563+0000 7f88d788c500 2 removed watcher, disabling cache 2023-09-22T13:17:59.501 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.564+0000 7f88d788c500 20 remove_watcher() i=1 2023-09-22T13:17:59.501 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.565+0000 7f88d788c500 20 remove_watcher() i=2 2023-09-22T13:17:59.501 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.565+0000 7f88d788c500 20 remove_watcher() i=3 2023-09-22T13:17:59.501 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.566+0000 7f88d788c500 20 remove_watcher() i=4 2023-09-22T13:17:59.501 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.567+0000 7f88d788c500 20 remove_watcher() i=5 2023-09-22T13:17:59.501 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.567+0000 7f88d788c500 20 remove_watcher() i=6 2023-09-22T13:17:59.502 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.568+0000 7f88d788c500 20 remove_watcher() i=7 2023-09-22T13:17:59.502 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.569+0000 7f8893fe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.502 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --fix --min-age-hours 0 --rgw-olh-pending-timeout-sec 0 --dump-keys 2023-09-22T13:17:59.502 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:59.502 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:59.521 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.612+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.521 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.614+0000 7ff8ff259500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.521 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.614+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.521 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.614+0000 7ff8ff259500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.522 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.614+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.522 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.614+0000 7ff8ff259500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.522 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.614+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.522 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.615+0000 7ff8ff259500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.522 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.625+0000 7ff8ff259500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:59.522 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.625+0000 7ff8ff259500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:59.522 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.625+0000 7ff8ff259500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:59.522 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.625+0000 7ff8ff259500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:59.522 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.625+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.522 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.625+0000 7ff8ff259500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.522 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.625+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.522 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.626+0000 7ff8ff259500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:59.523 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.626+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.523 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.626+0000 7ff8ff259500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.523 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.626+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.523 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.626+0000 7ff8ff259500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:59.523 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.626+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.523 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.626+0000 7ff8ff259500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.523 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.626+0000 7ff8ff259500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:59.523 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.626+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.523 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.626+0000 7ff8ff259500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.523 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.626+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.523 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.627+0000 7ff8ff259500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.523 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.627+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.524 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.627+0000 7ff8ff259500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:59.524 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.627+0000 7ff8ff259500 20 zonegroup default 2023-09-22T13:17:59.524 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.627+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.524 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.627+0000 7ff8ff259500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.524 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.627+0000 7ff8ff259500 10 Cannot find current period zone using local zone 2023-09-22T13:17:59.524 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.627+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.524 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.627+0000 7ff8ff259500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.524 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.627+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.524 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.628+0000 7ff8ff259500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.524 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.628+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.524 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.628+0000 7ff8ff259500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:59.525 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.628+0000 7ff8ff259500 20 zone default found 2023-09-22T13:17:59.525 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.628+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.525 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.628+0000 7ff8ff259500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.525 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.628+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.525 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.628+0000 7ff8ff259500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.525 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.628+0000 7ff8ff259500 20 started sync module instance, tier type = 2023-09-22T13:17:59.525 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.628+0000 7ff8ff259500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:59.525 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.635+0000 7ff8ff259500 20 add_watcher() i=0 2023-09-22T13:17:59.525 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.635+0000 7ff8ff259500 20 add_watcher() i=1 2023-09-22T13:17:59.525 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.635+0000 7ff8ff259500 20 add_watcher() i=2 2023-09-22T13:17:59.525 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.635+0000 7ff8ff259500 20 add_watcher() i=3 2023-09-22T13:17:59.525 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.635+0000 7ff8ff259500 20 add_watcher() i=4 2023-09-22T13:17:59.525 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.635+0000 7ff8ff259500 20 add_watcher() i=5 2023-09-22T13:17:59.526 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.635+0000 7ff8ff259500 20 add_watcher() i=6 2023-09-22T13:17:59.526 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.636+0000 7ff8ea73b700 0 -- 172.21.15.96:0/927733036 >> [v2:172.21.15.96:6807/140797,v1:172.21.15.96:6809/140797] conn(0x7ff8e003a900 msgr2=0x7ff8e002e400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:59.526 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.636+0000 7ff8ff259500 20 add_watcher() i=7 2023-09-22T13:17:59.526 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.636+0000 7ff8ff259500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:59.526 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.637+0000 7ff8bbfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.526 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.637+0000 7ff8ff259500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:59.526 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.637+0000 7ff8ff259500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:59.526 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.641+0000 7ff8ea73b700 0 -- 172.21.15.96:0/3126299222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611c758c160 msgr2=0x5611c7599850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-09-22T13:17:59.526 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.842+0000 7ff8ea73b700 0 --2- 172.21.15.96:0/3126299222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5611c758c160 0x5611c7599850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-09-22T13:17:59.526 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.844+0000 7ff8ff259500 5 note: GC not initialized 2023-09-22T13:17:59.526 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.845+0000 7ff82affd700 20 reqs_thread_entry: start 2023-09-22T13:17:59.527 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.845+0000 7ff828ff9700 20 reqs_thread_entry: start 2023-09-22T13:17:59.527 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.846+0000 7ff8ff259500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:59.527 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.846+0000 7ff8ff259500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:59.527 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.846+0000 7ff81effd700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:28 2023 2023-09-22T13:17:59.527 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:59.527 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.846+0000 7ff81e7fc700 20 reqs_thread_entry: start 2023-09-22T13:17:59.527 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.846+0000 7ff8ff259500 20 get_system_obj_state: rctx=0x7fff7852c3e8 obj=default.rgw.meta:root:check-bucket state=0x5611c73da810 s->prefetch_data=0 2023-09-22T13:17:59.527 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.846+0000 7ff8ff259500 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-09-22T13:17:59.527 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.846+0000 7ff8ff259500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x16 2023-09-22T13:17:59.527 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.846+0000 7ff8ff259500 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:59.527 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.846+0000 7ff8ff259500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:59.528 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.846+0000 7ff8ff259500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.528 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.846+0000 7ff8ff259500 10 cache get: name=default.rgw.meta+root+check-bucket : type miss (requested=0x11, cached=0x16) 2023-09-22T13:17:59.528 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.846+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.528 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:59.528 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-09-22T13:17:59.528 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 10 moving default.rgw.meta+root+check-bucket to cache LRU end 2023-09-22T13:17:59.528 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 20 get_system_obj_state: rctx=0x7fff7852c4b8 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x5611c73da810 s->prefetch_data=0 2023-09-22T13:17:59.528 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : miss 2023-09-22T13:17:59.528 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x16 2023-09-22T13:17:59.528 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:59.528 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:59.528 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:59.529 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.529 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.529 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.529 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 : type miss (requested=0x13, cached=0x16) 2023-09-22T13:17:59.529 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.529 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.529 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 info.flags=0x13 2023-09-22T13:17:59.529 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 10 moving default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 to cache LRU end 2023-09-22T13:17:59.529 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-09-22T13:17:59.529 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-09-22T13:17:59.529 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.847+0000 7ff8ff259500 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 2023-09-22T13:17:59.529 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.848+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.0 2023-09-22T13:17:59.530 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.848+0000 7ff8ff259500 1 NOTICE: finished shard 0 (0 entries removed) 2023-09-22T13:17:59.530 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.848+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:59.530 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.849+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.2 2023-09-22T13:17:59.530 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.849+0000 7ff8ff259500 1 NOTICE: finished shard 2 (0 entries removed) 2023-09-22T13:17:59.530 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.849+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.3 2023-09-22T13:17:59.530 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.850+0000 7ff8ff259500 1 NOTICE: finished shard 3 (0 entries removed) 2023-09-22T13:17:59.530 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.850+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.4 2023-09-22T13:17:59.530 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.850+0000 7ff8ff259500 1 NOTICE: finished shard 4 (0 entries removed) 2023-09-22T13:17:59.530 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.850+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:59.530 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.851+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.6 2023-09-22T13:17:59.530 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.851+0000 7ff8ff259500 1 NOTICE: finished shard 6 (0 entries removed) 2023-09-22T13:17:59.531 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.851+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.7 2023-09-22T13:17:59.531 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.852+0000 7ff8ff259500 1 NOTICE: finished shard 7 (0 entries removed) 2023-09-22T13:17:59.531 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.852+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:59.531 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.852+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.9 2023-09-22T13:17:59.531 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.853+0000 7ff8ff259500 1 NOTICE: finished shard 9 (0 entries removed) 2023-09-22T13:17:59.531 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.853+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:59.531 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.853+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:59.531 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.854+0000 7ff8ff259500 20 get_obj_state: rctx=0x5611c7628870 obj=check-bucket:e state=0x5611c764dd88 s->prefetch_data=0 2023-09-22T13:17:59.531 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.854+0000 7ff8ff259500 20 get_obj_state: setting s->obj_tag to yyncj686x8ogcxvee8sx5bf0wdtst93r 2023-09-22T13:17:59.531 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.854+0000 7ff8ff259500 20 get_obj_state_impl: setting s->olh_tag to ndg1sprkogvyc2vwof8mfd0ckyyd8wae 2023-09-22T13:17:59.531 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.854+0000 7ff8ff259500 20 bucket_index_guard_olh_op(): olh_state.olh_tag=ndg1sprkogvyc2vwof8mfd0ckyyd8wae 2023-09-22T13:17:59.532 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.855+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:59.532 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.856+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:59.532 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.856+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:59.532 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.857+0000 7ff8ff259500 20 olh_log_entry: epoch=2 op=2 key=e[] 2023-09-22T13:17:59.532 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.857+0000 7ff8ff259500 20 olh_log_entry: epoch=2 op=3 key=e[rpPC6CKT06oKWC.MbpstQ4rKuKrRJsB] 2023-09-22T13:17:59.532 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.857+0000 7ff8ff259500 20 get_obj_state: rctx=0x5611c7628870 obj=check-bucket:_:rpPC6CKT06oKWC.MbpstQ4rKuKrRJsB_e state=0x5611c7696e98 s->prefetch_data=0 2023-09-22T13:17:59.532 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.857+0000 7ff8ff259500 10 manifest: total_size = 22 2023-09-22T13:17:59.532 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.857+0000 7ff8ff259500 20 get_obj_state: setting s->obj_tag to 42123edd-4a97-43ec-80bc-fc5350f88c11.4527.17137161867192520781 2023-09-22T13:17:59.532 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.857+0000 7ff8ff259500 20 get_obj_state: rctx=0x5611c7628870 obj=check-bucket:_:rpPC6CKT06oKWC.MbpstQ4rKuKrRJsB_e state=0x5611c7696e98 s->prefetch_data=0 2023-09-22T13:17:59.532 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.857+0000 7ff8ff259500 20 prepare_atomic_modification: state is not atomic. state=0x5611c7696e98 2023-09-22T13:17:59.532 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.857+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:59.532 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.859+0000 7ff8ff259500 10 cache get: name=default.rgw.log++bucket.sync-source-hints.check-bucket : miss 2023-09-22T13:17:59.532 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.859+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.859+0000 7ff8ff259500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.859+0000 7ff8ff259500 10 cache put: name=default.rgw.log++bucket.sync-source-hints.check-bucket info.flags=0x0 2023-09-22T13:17:59.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.859+0000 7ff8ff259500 10 adding default.rgw.log++bucket.sync-source-hints.check-bucket to cache LRU end 2023-09-22T13:17:59.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.859+0000 7ff8ff259500 10 cache get: name=default.rgw.log++bucket.sync-target-hints.check-bucket : miss 2023-09-22T13:17:59.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.859+0000 7ff8ff259500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.860+0000 7ff8ff259500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.860+0000 7ff8ff259500 10 cache put: name=default.rgw.log++bucket.sync-target-hints.check-bucket info.flags=0x0 2023-09-22T13:17:59.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.860+0000 7ff8ff259500 10 adding default.rgw.log++bucket.sync-target-hints.check-bucket to cache LRU end 2023-09-22T13:17:59.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.860+0000 7ff8ff259500 10 chain_cache_entry: cache_locator= 2023-09-22T13:17:59.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.860+0000 7ff8ff259500 20 chain_cache_entry: couldn't find cache locator 2023-09-22T13:17:59.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.860+0000 7ff8ff259500 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-09-22T13:17:59.533 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.860+0000 7ff8ff259500 20 get_obj_state: rctx=0x5611c7628870 obj=check-bucket:e state=0x5611c764dd88 s->prefetch_data=0 2023-09-22T13:17:59.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.860+0000 7ff8ff259500 20 bucket_index_guard_olh_op(): olh_state.olh_tag=ndg1sprkogvyc2vwof8mfd0ckyyd8wae 2023-09-22T13:17:59.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.862+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:59.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.863+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.1 2023-09-22T13:17:59.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.863+0000 7ff8ff259500 10 cache get: name=default.rgw.log++bucket.sync-source-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:59.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.863+0000 7ff8ff259500 10 cache get: name=default.rgw.log++bucket.sync-target-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:59.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.863+0000 7ff8ff259500 10 chain_cache_entry: cache_locator= 2023-09-22T13:17:59.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.863+0000 7ff8ff259500 20 chain_cache_entry: couldn't find cache locator 2023-09-22T13:17:59.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.863+0000 7ff8ff259500 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-09-22T13:17:59.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.863+0000 7ff8ff259500 1 NOTICE: finished shard 1 (1 entries removed) 2023-09-22T13:17:59.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.863+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:59.534 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.863+0000 7ff8ff259500 20 get_obj_state: rctx=0x5611c764ca70 obj=check-bucket:d state=0x5611c764dd88 s->prefetch_data=0 2023-09-22T13:17:59.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.863+0000 7ff8ff259500 20 get_obj_state: setting s->obj_tag to 4g2h51wo5thkc4mbc35xx2n0692zxmjn 2023-09-22T13:17:59.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.863+0000 7ff8ff259500 20 get_obj_state_impl: setting s->olh_tag to nqpoo9m1d9z1c3irgak1dfwzcktarqge 2023-09-22T13:17:59.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.863+0000 7ff8ff259500 20 bucket_index_guard_olh_op(): olh_state.olh_tag=nqpoo9m1d9z1c3irgak1dfwzcktarqge 2023-09-22T13:17:59.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.864+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:59.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.865+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:59.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.865+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:59.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.866+0000 7ff8ff259500 20 olh_log_entry: epoch=3 op=3 key=d[w0VG6QIfEOXLKxOeQIPFj.JRCrI3AUU] 2023-09-22T13:17:59.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.866+0000 7ff8ff259500 20 get_obj_state: rctx=0x5611c764ca70 obj=check-bucket:_:w0VG6QIfEOXLKxOeQIPFj.JRCrI3AUU_d state=0x5611c769b278 s->prefetch_data=0 2023-09-22T13:17:59.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.866+0000 7ff8ff259500 10 manifest: total_size = 22 2023-09-22T13:17:59.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.866+0000 7ff8ff259500 20 get_obj_state: setting s->obj_tag to 42123edd-4a97-43ec-80bc-fc5350f88c11.4527.9604202517251727656 2023-09-22T13:17:59.535 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.866+0000 7ff8ff259500 20 get_obj_state: rctx=0x5611c764ca70 obj=check-bucket:_:w0VG6QIfEOXLKxOeQIPFj.JRCrI3AUU_d state=0x5611c769b278 s->prefetch_data=0 2023-09-22T13:17:59.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.866+0000 7ff8ff259500 20 prepare_atomic_modification: state is not atomic. state=0x5611c769b278 2023-09-22T13:17:59.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.866+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:59.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.868+0000 7ff8ff259500 10 cache get: name=default.rgw.log++bucket.sync-source-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:59.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.868+0000 7ff8ff259500 10 cache get: name=default.rgw.log++bucket.sync-target-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:59.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.868+0000 7ff8ff259500 10 chain_cache_entry: cache_locator= 2023-09-22T13:17:59.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.868+0000 7ff8ff259500 20 chain_cache_entry: couldn't find cache locator 2023-09-22T13:17:59.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.868+0000 7ff8ff259500 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-09-22T13:17:59.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.868+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:59.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.868+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:59.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.870+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.5 2023-09-22T13:17:59.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.870+0000 7ff8ff259500 10 cache get: name=default.rgw.log++bucket.sync-source-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:59.536 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.870+0000 7ff8ff259500 10 cache get: name=default.rgw.log++bucket.sync-target-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:59.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.870+0000 7ff8ff259500 10 chain_cache_entry: cache_locator= 2023-09-22T13:17:59.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.870+0000 7ff8ff259500 20 chain_cache_entry: couldn't find cache locator 2023-09-22T13:17:59.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.870+0000 7ff8ff259500 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-09-22T13:17:59.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.870+0000 7ff8ff259500 1 NOTICE: finished shard 5 (1 entries removed) 2023-09-22T13:17:59.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.870+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:59.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.870+0000 7ff8ff259500 20 get_obj_state: rctx=0x5611c766ed70 obj=check-bucket:c state=0x5611c764dc58 s->prefetch_data=0 2023-09-22T13:17:59.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.870+0000 7ff8ff259500 20 get_obj_state: setting s->obj_tag to 486c0rtp7uieu2yz20hracetgl3r5b1p 2023-09-22T13:17:59.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.870+0000 7ff8ff259500 20 get_obj_state_impl: setting s->olh_tag to 7p72bbbdpu00zauetgq7nzoxdphd7kvs 2023-09-22T13:17:59.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.870+0000 7ff8ff259500 20 bucket_index_guard_olh_op(): olh_state.olh_tag=7p72bbbdpu00zauetgq7nzoxdphd7kvs 2023-09-22T13:17:59.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.871+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:59.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.872+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:59.537 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.872+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:59.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.873+0000 7ff8ff259500 20 olh_log_entry: epoch=3 op=3 key=c[ZAIY0CMQk95j7L4dQ9fG94JOPMCJx0g] 2023-09-22T13:17:59.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.873+0000 7ff8ff259500 20 get_obj_state: rctx=0x5611c766ed70 obj=check-bucket:_:ZAIY0CMQk95j7L4dQ9fG94JOPMCJx0g_c state=0x5611c769b238 s->prefetch_data=0 2023-09-22T13:17:59.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.873+0000 7ff8ff259500 10 manifest: total_size = 22 2023-09-22T13:17:59.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.873+0000 7ff8ff259500 20 get_obj_state: setting s->obj_tag to 42123edd-4a97-43ec-80bc-fc5350f88c11.4527.11097331666955697361 2023-09-22T13:17:59.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.873+0000 7ff8ff259500 20 get_obj_state: rctx=0x5611c766ed70 obj=check-bucket:_:ZAIY0CMQk95j7L4dQ9fG94JOPMCJx0g_c state=0x5611c769b238 s->prefetch_data=0 2023-09-22T13:17:59.538 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.873+0000 7ff8ff259500 20 prepare_atomic_modification: state is not atomic. state=0x5611c769b238 2023-09-22T13:17:59.648 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.873+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:59.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.875+0000 7ff8ff259500 10 cache get: name=default.rgw.log++bucket.sync-source-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:59.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.875+0000 7ff8ff259500 10 cache get: name=default.rgw.log++bucket.sync-target-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:59.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.875+0000 7ff8ff259500 10 chain_cache_entry: cache_locator= 2023-09-22T13:17:59.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.875+0000 7ff8ff259500 20 chain_cache_entry: couldn't find cache locator 2023-09-22T13:17:59.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.875+0000 7ff8ff259500 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-09-22T13:17:59.649 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.876+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:59.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.876+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:59.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.877+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.8 2023-09-22T13:17:59.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.877+0000 7ff8ff259500 10 cache get: name=default.rgw.log++bucket.sync-source-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:59.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.877+0000 7ff8ff259500 10 cache get: name=default.rgw.log++bucket.sync-target-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:59.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.877+0000 7ff8ff259500 10 chain_cache_entry: cache_locator= 2023-09-22T13:17:59.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.877+0000 7ff8ff259500 20 chain_cache_entry: couldn't find cache locator 2023-09-22T13:17:59.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.877+0000 7ff8ff259500 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-09-22T13:17:59.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.877+0000 7ff8ff259500 1 NOTICE: finished shard 8 (1 entries removed) 2023-09-22T13:17:59.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.877+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:59.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.877+0000 7ff8ff259500 20 get_obj_state: rctx=0x5611c768ee70 obj=check-bucket:f state=0x5611c764dc58 s->prefetch_data=0 2023-09-22T13:17:59.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.878+0000 7ff8ff259500 20 get_obj_state: setting s->obj_tag to c3w77h3clzjnpok7i4xl6rt1vrrhdd2l 2023-09-22T13:17:59.650 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.878+0000 7ff8ff259500 20 get_obj_state_impl: setting s->olh_tag to 0pjuyfgt3tkf19uo4ll14jbgciuunpcx 2023-09-22T13:17:59.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.878+0000 7ff8ff259500 20 bucket_index_guard_olh_op(): olh_state.olh_tag=0pjuyfgt3tkf19uo4ll14jbgciuunpcx 2023-09-22T13:17:59.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.878+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:59.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.879+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:59.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.879+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:59.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.880+0000 7ff8ff259500 20 olh_log_entry: epoch=2 op=2 key=f[] 2023-09-22T13:17:59.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.880+0000 7ff8ff259500 20 olh_log_entry: epoch=2 op=3 key=f[0yriJ9izY94nUrC3f4D60B7ew3qRAVD] 2023-09-22T13:17:59.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.880+0000 7ff8ff259500 20 get_obj_state: rctx=0x5611c768ee70 obj=check-bucket:_:0yriJ9izY94nUrC3f4D60B7ew3qRAVD_f state=0x5611c762da38 s->prefetch_data=0 2023-09-22T13:17:59.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.880+0000 7ff8ff259500 10 manifest: total_size = 12 2023-09-22T13:17:59.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.880+0000 7ff8ff259500 20 get_obj_state: setting s->obj_tag to 42123edd-4a97-43ec-80bc-fc5350f88c11.4527.8703145652764977503 2023-09-22T13:17:59.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.880+0000 7ff8ff259500 20 get_obj_state: rctx=0x5611c768ee70 obj=check-bucket:_:0yriJ9izY94nUrC3f4D60B7ew3qRAVD_f state=0x5611c762da38 s->prefetch_data=0 2023-09-22T13:17:59.651 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.880+0000 7ff8ff259500 20 prepare_atomic_modification: state is not atomic. state=0x5611c762da38 2023-09-22T13:17:59.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.880+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:59.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.882+0000 7ff8ff259500 10 cache get: name=default.rgw.log++bucket.sync-source-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:59.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.882+0000 7ff8ff259500 10 cache get: name=default.rgw.log++bucket.sync-target-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:59.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.882+0000 7ff8ff259500 10 chain_cache_entry: cache_locator= 2023-09-22T13:17:59.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.882+0000 7ff8ff259500 20 chain_cache_entry: couldn't find cache locator 2023-09-22T13:17:59.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.882+0000 7ff8ff259500 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-09-22T13:17:59.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.883+0000 7ff8ff259500 20 get_obj_state: rctx=0x5611c768ee70 obj=check-bucket:f state=0x5611c764dc58 s->prefetch_data=0 2023-09-22T13:17:59.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.883+0000 7ff8ff259500 20 bucket_index_guard_olh_op(): olh_state.olh_tag=0pjuyfgt3tkf19uo4ll14jbgciuunpcx 2023-09-22T13:17:59.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.884+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:59.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.885+0000 7ff8ff259500 20 bucket index object: default.rgw.buckets.index:.dir.42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1.10 2023-09-22T13:17:59.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.885+0000 7ff8ff259500 10 cache get: name=default.rgw.log++bucket.sync-source-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:59.652 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.885+0000 7ff8ff259500 10 cache get: name=default.rgw.log++bucket.sync-target-hints.check-bucket : hit (negative entry) 2023-09-22T13:17:59.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.885+0000 7ff8ff259500 10 chain_cache_entry: cache_locator= 2023-09-22T13:17:59.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.885+0000 7ff8ff259500 20 chain_cache_entry: couldn't find cache locator 2023-09-22T13:17:59.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.885+0000 7ff8ff259500 20 couldn't put bucket_sync_policy cache entry, might have raced with data changes 2023-09-22T13:17:59.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.885+0000 7ff8ff259500 1 NOTICE: finished shard 10 (1 entries removed) 2023-09-22T13:17:59.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.885+0000 7ff8ff259500 1 NOTICE: finished all shards (4 entries removed) 2023-09-22T13:17:59.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.886+0000 7ff8ff259500 20 remove_watcher() i=0 2023-09-22T13:17:59.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.886+0000 7ff8ff259500 2 removed watcher, disabling cache 2023-09-22T13:17:59.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.887+0000 7ff8ff259500 20 remove_watcher() i=1 2023-09-22T13:17:59.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.888+0000 7ff8ff259500 20 remove_watcher() i=2 2023-09-22T13:17:59.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.889+0000 7ff8ff259500 20 remove_watcher() i=3 2023-09-22T13:17:59.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.890+0000 7ff8ff259500 20 remove_watcher() i=4 2023-09-22T13:17:59.653 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.890+0000 7ff8ff259500 20 remove_watcher() i=5 2023-09-22T13:17:59.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.891+0000 7ff8ff259500 20 remove_watcher() i=6 2023-09-22T13:17:59.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.892+0000 7ff8ff259500 20 remove_watcher() i=7 2023-09-22T13:17:59.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.893+0000 7ff8bbfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.654 INFO:tasks.workunit.client.0.smithi096.stderr:radosgw-admin bucket stats --bucket check-bucket 2023-09-22T13:17:59.654 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:17:59.654 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:17:59.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.980+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.981+0000 7fc70066b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.981+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.981+0000 7fc70066b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.981+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.654 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.982+0000 7fc70066b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.982+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.983+0000 7fc70066b500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.992+0000 7fc70066b500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:17:59.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.992+0000 7fc70066b500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:17:59.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.992+0000 7fc70066b500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:17:59.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.992+0000 7fc70066b500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:17:59.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.992+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.993+0000 7fc70066b500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.993+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.993+0000 7fc70066b500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:59.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.993+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.655 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.993+0000 7fc70066b500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.993+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.993+0000 7fc70066b500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:59.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.993+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.994+0000 7fc70066b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.994+0000 7fc70066b500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:17:59.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.994+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.994+0000 7fc70066b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.994+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.994+0000 7fc70066b500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.994+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.656 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.994+0000 7fc70066b500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:17:59.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.994+0000 7fc70066b500 20 zonegroup default 2023-09-22T13:17:59.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.994+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.995+0000 7fc70066b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.995+0000 7fc70066b500 10 Cannot find current period zone using local zone 2023-09-22T13:17:59.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.995+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.995+0000 7fc70066b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.995+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.995+0000 7fc70066b500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:17:59.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.995+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.996+0000 7fc70066b500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:17:59.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.996+0000 7fc70066b500 20 zone default found 2023-09-22T13:17:59.657 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.996+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.996+0000 7fc70066b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.996+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.996+0000 7fc70066b500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:17:59.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.996+0000 7fc70066b500 20 started sync module instance, tier type = 2023-09-22T13:17:59.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:57.996+0000 7fc70066b500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:17:59.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.003+0000 7fc70066b500 20 add_watcher() i=0 2023-09-22T13:17:59.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.003+0000 7fc70066b500 20 add_watcher() i=1 2023-09-22T13:17:59.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.003+0000 7fc70066b500 20 add_watcher() i=2 2023-09-22T13:17:59.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.003+0000 7fc70066b500 20 add_watcher() i=3 2023-09-22T13:17:59.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.003+0000 7fc70066b500 20 add_watcher() i=4 2023-09-22T13:17:59.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.003+0000 7fc70066b500 20 add_watcher() i=5 2023-09-22T13:17:59.658 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.003+0000 7fc70066b500 20 add_watcher() i=6 2023-09-22T13:17:59.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.004+0000 7fc70066b500 20 add_watcher() i=7 2023-09-22T13:17:59.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.004+0000 7fc70066b500 2 all 8 watchers are set, enabling cache 2023-09-22T13:17:59.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.005+0000 7fc6eb34c700 0 -- 172.21.15.96:0/874602758 >> [v2:172.21.15.96:6816/140810,v1:172.21.15.96:6817/140810] conn(0x562920306200 msgr2=0x5629203085f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:59.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.005+0000 7fc6bd7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.005+0000 7fc70066b500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:17:59.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.005+0000 7fc70066b500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:17:59.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.009+0000 7fc6ec34e700 0 -- 172.21.15.96:0/381854469 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562920342e40 msgr2=0x562920347ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:17:59.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.011+0000 7fc70066b500 5 note: GC not initialized 2023-09-22T13:17:59.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.012+0000 7fc62bfff700 20 reqs_thread_entry: start 2023-09-22T13:17:59.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.012+0000 7fc629ffb700 20 reqs_thread_entry: start 2023-09-22T13:17:59.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.012+0000 7fc70066b500 20 init_complete bucket index max shards: 11 2023-09-22T13:17:59.659 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.013+0000 7fc70066b500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:17:59.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.013+0000 7fc617fff700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:18:28 2023 2023-09-22T13:17:59.660 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:17:59.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.013+0000 7fc6177fe700 20 reqs_thread_entry: start 2023-09-22T13:17:59.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.013+0000 7fc70066b500 20 get_system_obj_state: rctx=0x7ffce671f848 obj=default.rgw.meta:root:check-bucket state=0x562920199810 s->prefetch_data=0 2023-09-22T13:17:59.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.013+0000 7fc70066b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.013+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.013+0000 7fc70066b500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:59.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.013+0000 7fc70066b500 20 get_system_obj_state: rctx=0x7ffce671f918 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562920199810 s->prefetch_data=0 2023-09-22T13:17:59.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.014+0000 7fc70066b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.014+0000 7fc70066b500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.014+0000 7fc70066b500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.660 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.014+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.014+0000 7fc70066b500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.014+0000 7fc70066b500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.014+0000 7fc70066b500 20 get_system_obj_state: rctx=0x7ffce671fc10 obj=default.rgw.meta:root:check-bucket state=0x562920199810 s->prefetch_data=0 2023-09-22T13:17:59.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.015+0000 7fc70066b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.015+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.015+0000 7fc70066b500 20 rados_obj.operate() r=0 bl.length=190 2023-09-22T13:17:59.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.015+0000 7fc70066b500 20 get_system_obj_state: rctx=0x7ffce671fc10 obj=default.rgw.meta:root:.bucket.meta.check-bucket:42123edd-4a97-43ec-80bc-fc5350f88c11.4533.1 state=0x562920153910 s->prefetch_data=0 2023-09-22T13:17:59.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.015+0000 7fc70066b500 20 get_system_obj_state: s->obj_tag was set empty 2023-09-22T13:17:59.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.015+0000 7fc70066b500 20 Read xattr: user.rgw.acl 2023-09-22T13:17:59.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.015+0000 7fc70066b500 20 Read xattr: user.rgw.idtag 2023-09-22T13:17:59.661 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.015+0000 7fc70066b500 20 rados->read ofs=0 len=0 2023-09-22T13:17:59.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.016+0000 7fc70066b500 20 rados_obj.operate() r=0 bl.length=368 2023-09-22T13:17:59.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.016+0000 7fc70066b500 20 couldn't put binfo cache entry, might have raced with data changes 2023-09-22T13:17:59.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.017+0000 7fc70066b500 20 remove_watcher() i=0 2023-09-22T13:17:59.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.017+0000 7fc70066b500 2 removed watcher, disabling cache 2023-09-22T13:17:59.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.018+0000 7fc70066b500 20 remove_watcher() i=1 2023-09-22T13:17:59.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.019+0000 7fc70066b500 20 remove_watcher() i=2 2023-09-22T13:17:59.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.020+0000 7fc70066b500 20 remove_watcher() i=3 2023-09-22T13:17:59.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.020+0000 7fc70066b500 20 remove_watcher() i=4 2023-09-22T13:17:59.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.021+0000 7fc70066b500 20 remove_watcher() i=5 2023-09-22T13:17:59.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.022+0000 7fc70066b500 20 remove_watcher() i=6 2023-09-22T13:17:59.662 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.023+0000 7fc70066b500 20 remove_watcher() i=7 2023-09-22T13:17:59.663 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:17:58.023+0000 7fc6bd7e2700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:17:59.663 INFO:tasks.workunit.client.0.smithi096.stderr:{'rgw.main': {'size': 0, 'size_actual': 0, 'size_utilized': 0, 'size_kb': 0, 'size_kb_actual': 0, 'size_kb_utilized': 0, 'num_objects': 0}} 2023-09-22T13:17:59.663 INFO:tasks.workunit.client.0.smithi096.stderr:Deleting bucket check-bucket 2023-09-22T13:17:59.663 INFO:tasks.workunit.client.0.smithi096.stderr:Completed bucket check tests 2023-09-22T13:17:59.663 INFO:tasks.workunit.client.0.smithi096.stdout:connected to http://localhost:80 2023-09-22T13:17:59.665 INFO:tasks.workunit.client.0.smithi096.stderr:+ deactivate 2023-09-22T13:17:59.665 INFO:tasks.workunit.client.0.smithi096.stderr:+ '[' -n /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin ']' 2023-09-22T13:17:59.665 INFO:tasks.workunit.client.0.smithi096.stderr:+ PATH=/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-09-22T13:17:59.665 INFO:tasks.workunit.client.0.smithi096.stderr:+ export PATH 2023-09-22T13:17:59.665 INFO:tasks.workunit.client.0.smithi096.stderr:+ unset _OLD_VIRTUAL_PATH 2023-09-22T13:17:59.665 INFO:tasks.workunit.client.0.smithi096.stderr:+ '[' -n '' ']' 2023-09-22T13:17:59.665 INFO:tasks.workunit.client.0.smithi096.stderr:+ '[' -n /usr/bin/bash -o -n '' ']' 2023-09-22T13:17:59.665 INFO:tasks.workunit.client.0.smithi096.stderr:+ hash -r 2023-09-22T13:17:59.665 INFO:tasks.workunit.client.0.smithi096.stderr:+ '[' -n '' ']' 2023-09-22T13:17:59.665 INFO:tasks.workunit.client.0.smithi096.stderr:+ unset VIRTUAL_ENV 2023-09-22T13:17:59.665 INFO:tasks.workunit.client.0.smithi096.stderr:+ '[' '!' '' = nondestructive ']' 2023-09-22T13:17:59.666 INFO:tasks.workunit.client.0.smithi096.stderr:+ unset -f deactivate 2023-09-22T13:17:59.666 INFO:tasks.workunit.client.0.smithi096.stderr:+ echo OK. 2023-09-22T13:17:59.666 INFO:tasks.workunit.client.0.smithi096.stdout:OK. 2023-09-22T13:17:59.667 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-09-22T13:17:59.667 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-22T13:17:59.701 INFO:tasks.workunit:Stopping ['rgw/run-bucket-check.sh'] on client.0... 2023-09-22T13:17:59.701 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-09-22T13:18:00.030 DEBUG:teuthology.parallel:result is None 2023-09-22T13:18:00.030 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-09-22T13:18:00.065 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-09-22T13:18:00.065 DEBUG:teuthology.orchestra.run.smithi096:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-09-22T13:18:00.163 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-09-22T13:18:00.163 INFO:teuthology.run_tasks:Running task workunit... 2023-09-22T13:18:00.174 INFO:tasks.workunit:Pulling workunits from ref 31c126b7c1c0c06b66d9d19c5bfc51c46b503392 2023-09-22T13:18:00.174 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-09-22T13:18:00.174 DEBUG:teuthology.orchestra.run.smithi096:> stat -- /home/ubuntu/cephtest/mnt.0 2023-09-22T13:18:00.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-22T13:18:00.192 INFO:teuthology.orchestra.run.smithi096.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-09-22T13:18:00.192 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-09-22T13:18:00.250 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-09-22T13:18:00.250 DEBUG:teuthology.orchestra.run.smithi096:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-09-22T13:18:00.307 INFO:tasks.workunit:timeout=3h 2023-09-22T13:18:00.308 INFO:tasks.workunit:cleanup=True 2023-09-22T13:18:00.308 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 31c126b7c1c0c06b66d9d19c5bfc51c46b503392 2023-09-22T13:18:00.366 INFO:tasks.workunit.client.0.smithi096.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-09-22T13:19:00.107 INFO:tasks.workunit.client.0.smithi096.stderr:Note: switching to '31c126b7c1c0c06b66d9d19c5bfc51c46b503392'. 2023-09-22T13:19:00.107 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:19:00.107 INFO:tasks.workunit.client.0.smithi096.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-09-22T13:19:00.107 INFO:tasks.workunit.client.0.smithi096.stderr:changes and commit them, and you can discard any commits you make in this 2023-09-22T13:19:00.107 INFO:tasks.workunit.client.0.smithi096.stderr:state without impacting any branches by switching back to a branch. 2023-09-22T13:19:00.107 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:19:00.108 INFO:tasks.workunit.client.0.smithi096.stderr:If you want to create a new branch to retain commits you create, you may 2023-09-22T13:19:00.108 INFO:tasks.workunit.client.0.smithi096.stderr:do so (now or later) by using -c with the switch command. Example: 2023-09-22T13:19:00.108 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:19:00.108 INFO:tasks.workunit.client.0.smithi096.stderr: git switch -c 2023-09-22T13:19:00.108 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:19:00.108 INFO:tasks.workunit.client.0.smithi096.stderr:Or undo this operation with: 2023-09-22T13:19:00.108 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:19:00.108 INFO:tasks.workunit.client.0.smithi096.stderr: git switch - 2023-09-22T13:19:00.108 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:19:00.108 INFO:tasks.workunit.client.0.smithi096.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-09-22T13:19:00.108 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:19:00.108 INFO:tasks.workunit.client.0.smithi096.stderr:HEAD is now at 31c126b7c1c rgw: radosgw-admin bucket check should only print index entries with --check-objects flag 2023-09-22T13:19:00.113 DEBUG:teuthology.orchestra.run.smithi096:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-09-22T13:19:00.171 INFO:tasks.workunit.client.0.smithi096.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-09-22T13:19:00.173 INFO:tasks.workunit.client.0.smithi096.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-09-22T13:19:00.173 INFO:tasks.workunit.client.0.smithi096.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-09-22T13:19:00.220 INFO:tasks.workunit.client.0.smithi096.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-09-22T13:19:00.266 INFO:tasks.workunit.client.0.smithi096.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-09-22T13:19:00.303 INFO:tasks.workunit.client.0.smithi096.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-09-22T13:19:00.304 INFO:tasks.workunit.client.0.smithi096.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-09-22T13:19:00.305 INFO:tasks.workunit.client.0.smithi096.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-09-22T13:19:00.341 INFO:tasks.workunit.client.0.smithi096.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-09-22T13:19:00.344 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:19:00.344 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-09-22T13:19:00.402 INFO:tasks.workunit:Running workunits matching cls/test_cls_lock.sh on client.0... 2023-09-22T13:19:00.403 INFO:tasks.workunit:Running workunit cls/test_cls_lock.sh... 2023-09-22T13:19:00.404 DEBUG:teuthology.orchestra.run.smithi096:workunit test cls/test_cls_lock.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=31c126b7c1c0c06b66d9d19c5bfc51c46b503392 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh 2023-09-22T13:19:00.473 INFO:tasks.workunit.client.0.smithi096.stdout:Running main() from gmock_main.cc 2023-09-22T13:19:00.473 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 11 tests from 1 test suite. 2023-09-22T13:19:00.473 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:19:00.473 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 11 tests from ClsLock 2023-09-22T13:19:00.473 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] ClsLock.TestMultiLocking 2023-09-22T13:19:01.852 INFO:tasks.workunit.client.0.smithi096.stdout:lock: mylock 2023-09-22T13:19:01.853 INFO:tasks.workunit.client.0.smithi096.stdout: lock_type: exclusive 2023-09-22T13:19:01.853 INFO:tasks.workunit.client.0.smithi096.stdout: tag: 2023-09-22T13:19:01.853 INFO:tasks.workunit.client.0.smithi096.stdout: lockers: 2023-09-22T13:19:01.853 INFO:tasks.workunit.client.0.smithi096.stdout: client.4790 expiration=2023-09-22T13:21:01.850364+0000 addr=v1:172.21.15.96:0/1770717508 cookie= 2023-09-22T13:19:01.855 INFO:tasks.workunit.client.0.smithi096.stdout:lock: mylock 2023-09-22T13:19:01.856 INFO:tasks.workunit.client.0.smithi096.stdout: lock_type: shared 2023-09-22T13:19:01.856 INFO:tasks.workunit.client.0.smithi096.stdout: tag: 2023-09-22T13:19:01.856 INFO:tasks.workunit.client.0.smithi096.stdout: lockers: 2023-09-22T13:19:01.856 INFO:tasks.workunit.client.0.smithi096.stdout: client.4790 expiration=2023-09-22T13:21:01.855632+0000 addr=v1:172.21.15.96:0/1770717508 cookie= 2023-09-22T13:19:01.856 INFO:tasks.workunit.client.0.smithi096.stdout: client.4977 expiration=0.000000 addr=v1:172.21.15.96:0/546928463 cookie= 2023-09-22T13:19:01.856 INFO:tasks.workunit.client.0.smithi096.stdout:lock: mylock 2023-09-22T13:19:01.856 INFO:tasks.workunit.client.0.smithi096.stdout: lock_type: shared 2023-09-22T13:19:01.857 INFO:tasks.workunit.client.0.smithi096.stdout: tag: 2023-09-22T13:19:01.857 INFO:tasks.workunit.client.0.smithi096.stdout: lockers: 2023-09-22T13:19:01.857 INFO:tasks.workunit.client.0.smithi096.stdout: client.4977 expiration=0.000000 addr=v1:172.21.15.96:0/546928463 cookie= 2023-09-22T13:19:02.750 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] ClsLock.TestMultiLocking (2277 ms) 2023-09-22T13:19:02.750 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] ClsLock.TestMeta 2023-09-22T13:19:04.807 INFO:tasks.workunit.client.0.smithi096.stdout:lock: mylock 2023-09-22T13:19:04.808 INFO:tasks.workunit.client.0.smithi096.stdout: lock_type: shared 2023-09-22T13:19:04.808 INFO:tasks.workunit.client.0.smithi096.stdout: tag: 2023-09-22T13:19:04.808 INFO:tasks.workunit.client.0.smithi096.stdout: lockers: 2023-09-22T13:19:04.808 INFO:tasks.workunit.client.0.smithi096.stdout: client.4805 expiration=0.000000 addr=v1:172.21.15.96:0/1101113981 cookie= 2023-09-22T13:19:04.809 INFO:tasks.workunit.client.0.smithi096.stdout:lock: mylock 2023-09-22T13:19:04.809 INFO:tasks.workunit.client.0.smithi096.stdout: lock_type: exclusive 2023-09-22T13:19:04.809 INFO:tasks.workunit.client.0.smithi096.stdout: tag: new_tag 2023-09-22T13:19:04.809 INFO:tasks.workunit.client.0.smithi096.stdout: lockers: 2023-09-22T13:19:04.810 INFO:tasks.workunit.client.0.smithi096.stdout: client.4799 expiration=0.000000 addr=v1:172.21.15.96:0/41218557 cookie= 2023-09-22T13:19:05.775 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] ClsLock.TestMeta (3024 ms) 2023-09-22T13:19:05.775 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] ClsLock.TestCookie 2023-09-22T13:19:07.832 INFO:tasks.workunit.client.0.smithi096.stdout:lock: mylock 2023-09-22T13:19:07.832 INFO:tasks.workunit.client.0.smithi096.stdout: lock_type: exclusive 2023-09-22T13:19:07.832 INFO:tasks.workunit.client.0.smithi096.stdout: tag: 2023-09-22T13:19:07.832 INFO:tasks.workunit.client.0.smithi096.stdout: lockers: 2023-09-22T13:19:07.833 INFO:tasks.workunit.client.0.smithi096.stdout:lock: mylock 2023-09-22T13:19:07.834 INFO:tasks.workunit.client.0.smithi096.stdout: lock_type: shared 2023-09-22T13:19:07.834 INFO:tasks.workunit.client.0.smithi096.stdout: tag: 2023-09-22T13:19:07.834 INFO:tasks.workunit.client.0.smithi096.stdout: lockers: 2023-09-22T13:19:07.834 INFO:tasks.workunit.client.0.smithi096.stdout: client.4998 expiration=0.000000 addr=v1:172.21.15.96:0/349321216 cookie= 2023-09-22T13:19:07.834 INFO:tasks.workunit.client.0.smithi096.stdout: client.4998 expiration=0.000000 addr=v1:172.21.15.96:0/349321216 cookie=new cookie 2023-09-22T13:19:08.784 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] ClsLock.TestCookie (3009 ms) 2023-09-22T13:19:08.784 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] ClsLock.TestMultipleLocks 2023-09-22T13:19:11.827 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] ClsLock.TestMultipleLocks (3044 ms) 2023-09-22T13:19:11.827 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] ClsLock.TestLockDuration 2023-09-22T13:19:19.866 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] ClsLock.TestLockDuration (8038 ms) 2023-09-22T13:19:19.866 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] ClsLock.TestAssertLocked 2023-09-22T13:19:22.885 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] ClsLock.TestAssertLocked (3020 ms) 2023-09-22T13:19:22.885 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] ClsLock.TestSetCookie 2023-09-22T13:19:25.909 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] ClsLock.TestSetCookie (3023 ms) 2023-09-22T13:19:25.909 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] ClsLock.TestRenew 2023-09-22T13:19:46.683 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] ClsLock.TestRenew (20774 ms) 2023-09-22T13:19:46.683 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralBasic 2023-09-22T13:19:53.720 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] ClsLock.TestExclusiveEphemeralBasic (7037 ms) 2023-09-22T13:19:53.720 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealEphemeral 2023-09-22T13:20:01.778 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealEphemeral (8058 ms) 2023-09-22T13:20:01.779 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealExclusive 2023-09-22T13:20:09.820 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealExclusive (8041 ms) 2023-09-22T13:20:09.820 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 11 tests from ClsLock (69346 ms total) 2023-09-22T13:20:09.820 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:20:09.820 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:20:09.820 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 11 tests from 1 test suite ran. (69346 ms total) 2023-09-22T13:20:09.820 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 11 tests. 2023-09-22T13:20:09.823 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-09-22T13:20:09.823 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-22T13:20:09.898 INFO:tasks.workunit:Running workunits matching cls/test_cls_log.sh on client.0... 2023-09-22T13:20:09.899 INFO:tasks.workunit:Running workunit cls/test_cls_log.sh... 2023-09-22T13:20:09.899 DEBUG:teuthology.orchestra.run.smithi096:workunit test cls/test_cls_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=31c126b7c1c0c06b66d9d19c5bfc51c46b503392 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_log.sh 2023-09-22T13:20:09.972 INFO:tasks.workunit.client.0.smithi096.stdout:Running main() from gmock_main.cc 2023-09-22T13:20:09.972 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 4 tests from 1 test suite. 2023-09-22T13:20:09.972 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:20:09.972 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 4 tests from cls_log 2023-09-22T13:20:09.972 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_log.test_log_add_same_time 2023-09-22T13:20:12.836 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_log.test_log_add_same_time (2864 ms) 2023-09-22T13:20:12.836 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_log.test_log_add_different_time 2023-09-22T13:20:15.864 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_log.test_log_add_different_time (3029 ms) 2023-09-22T13:20:15.864 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_log.trim_by_time 2023-09-22T13:20:18.887 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_log.trim_by_time (3022 ms) 2023-09-22T13:20:18.887 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_log.trim_by_marker 2023-09-22T13:20:21.924 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_log.trim_by_marker (3037 ms) 2023-09-22T13:20:21.924 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 4 tests from cls_log (11952 ms total) 2023-09-22T13:20:21.925 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:20:21.925 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:20:21.925 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 4 tests from 1 test suite ran. (11952 ms total) 2023-09-22T13:20:21.925 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 4 tests. 2023-09-22T13:20:21.927 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-09-22T13:20:21.927 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-22T13:20:21.999 INFO:tasks.workunit:Running workunits matching cls/test_cls_refcount.sh on client.0... 2023-09-22T13:20:22.000 INFO:tasks.workunit:Running workunit cls/test_cls_refcount.sh... 2023-09-22T13:20:22.000 DEBUG:teuthology.orchestra.run.smithi096:workunit test cls/test_cls_refcount.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=31c126b7c1c0c06b66d9d19c5bfc51c46b503392 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_refcount.sh 2023-09-22T13:20:22.075 INFO:tasks.workunit.client.0.smithi096.stdout:Running main() from gmock_main.cc 2023-09-22T13:20:22.075 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 10 tests from 1 test suite. 2023-09-22T13:20:22.076 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:20:22.076 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 10 tests from cls_rgw 2023-09-22T13:20:22.076 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.test_implicit 2023-09-22T13:20:24.949 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.test_implicit (2875 ms) 2023-09-22T13:20:24.950 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.test_implicit_idempotent 2023-09-22T13:20:27.964 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.test_implicit_idempotent (3014 ms) 2023-09-22T13:20:27.964 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.test_put_snap 2023-09-22T13:20:36.654 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.test_put_snap (8690 ms) 2023-09-22T13:20:36.654 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.test_explicit 2023-09-22T13:20:39.669 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.test_explicit (3016 ms) 2023-09-22T13:20:39.670 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.set 2023-09-22T13:20:42.691 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.set (3020 ms) 2023-09-22T13:20:42.691 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.test_implicit_ec 2023-09-22T13:20:49.742 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.test_implicit_ec (7052 ms) 2023-09-22T13:20:49.743 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.test_implicit_idempotent_ec 2023-09-22T13:20:56.774 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.test_implicit_idempotent_ec (7031 ms) 2023-09-22T13:20:56.774 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.test_put_snap_ec 2023-09-22T13:21:09.707 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.test_put_snap_ec (12933 ms) 2023-09-22T13:21:09.707 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.test_explicit_ec 2023-09-22T13:21:16.733 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.test_explicit_ec (7026 ms) 2023-09-22T13:21:16.733 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.set_ec 2023-09-22T13:21:24.799 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.set_ec (8066 ms) 2023-09-22T13:21:24.799 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 10 tests from cls_rgw (62724 ms total) 2023-09-22T13:21:24.799 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:21:24.799 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:21:24.799 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 10 tests from 1 test suite ran. (62724 ms total) 2023-09-22T13:21:24.799 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 10 tests. 2023-09-22T13:21:24.801 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-09-22T13:21:24.801 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-22T13:21:24.874 INFO:tasks.workunit:Running workunits matching cls/test_cls_rgw.sh on client.0... 2023-09-22T13:21:24.875 INFO:tasks.workunit:Running workunit cls/test_cls_rgw.sh... 2023-09-22T13:21:24.875 DEBUG:teuthology.orchestra.run.smithi096:workunit test cls/test_cls_rgw.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=31c126b7c1c0c06b66d9d19c5bfc51c46b503392 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh 2023-09-22T13:21:24.949 INFO:tasks.workunit.client.0.smithi096.stdout:Running main() from gmock_main.cc 2023-09-22T13:21:24.950 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 16 tests from 1 test suite. 2023-09-22T13:21:24.950 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:21:24.950 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 16 tests from cls_rgw 2023-09-22T13:21:26.820 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.index_basic 2023-09-22T13:21:26.861 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.index_basic (40 ms) 2023-09-22T13:21:26.861 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.index_multiple_obj_writers 2023-09-22T13:21:26.881 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.index_multiple_obj_writers (21 ms) 2023-09-22T13:21:26.881 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.index_remove_object 2023-09-22T13:21:26.916 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.index_remove_object (34 ms) 2023-09-22T13:21:26.916 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.index_suggest 2023-09-22T13:21:28.321 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.index_suggest (1406 ms) 2023-09-22T13:21:28.322 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.index_suggest_complete 2023-09-22T13:21:28.326 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.index_suggest_complete (5 ms) 2023-09-22T13:21:28.326 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.index_list 2023-09-22T13:21:28.344 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.index_list (18 ms) 2023-09-22T13:21:28.344 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.index_list_delimited 2023-09-22T13:21:43.464 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.index_list_delimited (15120 ms) 2023-09-22T13:21:43.465 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.bi_list 2023-09-22T13:21:43.522 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.bi_list (58 ms) 2023-09-22T13:21:43.523 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.gc_set 2023-09-22T13:21:43.531 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.gc_set (9 ms) 2023-09-22T13:21:43.532 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.gc_list 2023-09-22T13:21:43.541 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.gc_list (9 ms) 2023-09-22T13:21:43.541 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.gc_defer 2023-09-22T13:21:50.889 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.gc_defer (7349 ms) 2023-09-22T13:21:50.889 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.usage_basic 2023-09-22T13:21:51.017 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.usage_basic (128 ms) 2023-09-22T13:21:51.017 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.usage_clear_no_obj 2023-09-22T13:21:51.018 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.usage_clear_no_obj (1 ms) 2023-09-22T13:21:51.018 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.usage_clear 2023-09-22T13:21:51.094 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.usage_clear (76 ms) 2023-09-22T13:21:51.094 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.bi_log_trim 2023-09-22T13:21:51.664 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.bi_log_trim (569 ms) 2023-09-22T13:21:51.664 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] cls_rgw.index_racing_removes 2023-09-22T13:21:51.675 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] cls_rgw.index_racing_removes (11 ms) 2023-09-22T13:21:51.889 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 16 tests from cls_rgw (24854 ms total) 2023-09-22T13:21:51.889 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:21:51.889 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:21:51.889 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 16 tests from 1 test suite ran. (26939 ms total) 2023-09-22T13:21:51.890 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 16 tests. 2023-09-22T13:21:51.891 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-09-22T13:21:51.891 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-22T13:21:51.959 INFO:tasks.workunit:Running workunits matching cls/test_cls_cmpomap.sh on client.0... 2023-09-22T13:21:51.960 INFO:tasks.workunit:Running workunit cls/test_cls_cmpomap.sh... 2023-09-22T13:21:51.960 DEBUG:teuthology.orchestra.run.smithi096:workunit test cls/test_cls_cmpomap.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=31c126b7c1c0c06b66d9d19c5bfc51c46b503392 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_cmpomap.sh 2023-09-22T13:21:52.030 INFO:tasks.workunit.client.0.smithi096.stdout:Running main() from gmock_main.cc 2023-09-22T13:21:52.031 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 31 tests from 1 test suite. 2023-09-22T13:21:52.031 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:21:53.906 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 31 tests from CmpOmap 2023-09-22T13:21:53.906 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_str 2023-09-22T13:21:53.926 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_vals_noexist_str (20 ms) 2023-09-22T13:21:53.926 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_str_default 2023-09-22T13:21:53.961 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_vals_noexist_str_default (34 ms) 2023-09-22T13:21:53.961 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_u64 2023-09-22T13:21:53.964 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_vals_noexist_u64 (4 ms) 2023-09-22T13:21:53.964 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_u64_default 2023-09-22T13:21:53.966 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_vals_noexist_u64_default (2 ms) 2023-09-22T13:21:53.967 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_vals_str 2023-09-22T13:21:53.973 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_vals_str (7 ms) 2023-09-22T13:21:53.974 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_vals_u64 2023-09-22T13:21:53.979 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_vals_u64 (6 ms) 2023-09-22T13:21:53.980 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_input 2023-09-22T13:21:53.982 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_input (3 ms) 2023-09-22T13:21:53.982 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_default 2023-09-22T13:21:53.984 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_default (2 ms) 2023-09-22T13:21:53.985 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_vals_u64_empty_default 2023-09-22T13:21:53.987 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_vals_u64_empty_default (3 ms) 2023-09-22T13:21:53.987 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_value 2023-09-22T13:21:53.991 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_value (4 ms) 2023-09-22T13:21:53.991 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_vals_at_max_keys 2023-09-22T13:21:53.991 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_vals_at_max_keys (0 ms) 2023-09-22T13:21:53.992 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_vals_over_max_keys 2023-09-22T13:21:53.992 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_vals_over_max_keys (1 ms) 2023-09-22T13:21:53.992 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_str 2023-09-22T13:21:53.997 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_str (5 ms) 2023-09-22T13:21:53.997 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_str_default 2023-09-22T13:21:54.002 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_str_default (5 ms) 2023-09-22T13:21:54.002 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_u64 2023-09-22T13:21:54.007 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_u64 (5 ms) 2023-09-22T13:21:54.007 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_u64_default 2023-09-22T13:21:54.012 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_u64_default (5 ms) 2023-09-22T13:21:54.012 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_set_vals_str 2023-09-22T13:21:54.019 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_set_vals_str (6 ms) 2023-09-22T13:21:54.019 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64 2023-09-22T13:21:54.025 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_set_vals_u64 (6 ms) 2023-09-22T13:21:54.025 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64_einval 2023-09-22T13:21:54.027 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_set_vals_u64_einval (2 ms) 2023-09-22T13:21:54.027 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64_eio 2023-09-22T13:21:54.028 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_set_vals_u64_eio (2 ms) 2023-09-22T13:21:54.029 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_set_vals_at_max_keys 2023-09-22T13:21:54.029 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_set_vals_at_max_keys (1 ms) 2023-09-22T13:21:54.030 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_set_vals_over_max_keys 2023-09-22T13:21:54.030 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_set_vals_over_max_keys (1 ms) 2023-09-22T13:21:54.030 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_rm_keys_noexist_str 2023-09-22T13:21:54.035 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_rm_keys_noexist_str (5 ms) 2023-09-22T13:21:54.035 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_rm_keys_noexist_u64 2023-09-22T13:21:54.039 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_rm_keys_noexist_u64 (4 ms) 2023-09-22T13:21:54.039 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_rm_keys_str 2023-09-22T13:21:54.045 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_rm_keys_str (6 ms) 2023-09-22T13:21:54.045 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64 2023-09-22T13:21:54.051 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64 (6 ms) 2023-09-22T13:21:54.051 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_einval 2023-09-22T13:21:54.052 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_einval (1 ms) 2023-09-22T13:21:54.052 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_eio 2023-09-22T13:21:54.054 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_eio (2 ms) 2023-09-22T13:21:54.054 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_rm_keys_at_max_keys 2023-09-22T13:21:54.055 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_rm_keys_at_max_keys (1 ms) 2023-09-22T13:21:54.055 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_rm_keys_over_max_keys 2023-09-22T13:21:54.056 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_rm_keys_over_max_keys (1 ms) 2023-09-22T13:21:54.056 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_empty 2023-09-22T13:21:54.061 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_empty (5 ms) 2023-09-22T13:21:54.062 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 31 tests from CmpOmap (155 ms total) 2023-09-22T13:21:54.062 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:21:54.062 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:21:54.906 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 31 tests from 1 test suite ran. (2877 ms total) 2023-09-22T13:21:54.906 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 31 tests. 2023-09-22T13:21:54.908 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-09-22T13:21:54.908 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-22T13:21:54.977 INFO:tasks.workunit:Running workunits matching cls/test_cls_2pc_queue.sh on client.0... 2023-09-22T13:21:54.978 INFO:tasks.workunit:Running workunit cls/test_cls_2pc_queue.sh... 2023-09-22T13:21:54.978 DEBUG:teuthology.orchestra.run.smithi096:workunit test cls/test_cls_2pc_queue.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=31c126b7c1c0c06b66d9d19c5bfc51c46b503392 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2023-09-22T13:21:55.048 INFO:tasks.workunit.client.0.smithi096.stdout:Running main() from gmock_main.cc 2023-09-22T13:21:55.048 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 22 tests from 1 test suite. 2023-09-22T13:21:55.048 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:21:55.048 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 22 tests from TestCls2PCQueue 2023-09-22T13:21:55.048 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.GetCapacity 2023-09-22T13:21:57.948 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.GetCapacity (2900 ms) 2023-09-22T13:21:57.949 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.AsyncGetCapacity 2023-09-22T13:22:00.964 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.AsyncGetCapacity (3015 ms) 2023-09-22T13:22:00.964 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.Reserve 2023-09-22T13:22:03.989 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.Reserve (3026 ms) 2023-09-22T13:22:03.990 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.AsyncReserve 2023-09-22T13:22:07.028 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.AsyncReserve (3039 ms) 2023-09-22T13:22:07.028 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.Commit 2023-09-22T13:22:10.055 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.Commit (3023 ms) 2023-09-22T13:22:10.055 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.Abort 2023-09-22T13:22:13.074 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.Abort (3022 ms) 2023-09-22T13:22:13.074 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.ReserveError 2023-09-22T13:22:16.078 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.ReserveError (3005 ms) 2023-09-22T13:22:16.079 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.CommitError 2023-09-22T13:22:19.094 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.CommitError (3014 ms) 2023-09-22T13:22:19.094 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.AbortError 2023-09-22T13:22:22.115 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.AbortError (3022 ms) 2023-09-22T13:22:22.115 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.MultiReserve 2023-09-22T13:22:25.128 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.MultiReserve (3013 ms) 2023-09-22T13:22:25.128 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.MultiCommit 2023-09-22T13:22:28.150 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.MultiCommit (3023 ms) 2023-09-22T13:22:28.151 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.MultiAbort 2023-09-22T13:22:31.163 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.MultiAbort (3013 ms) 2023-09-22T13:22:31.164 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.ReserveCommit 2023-09-22T13:22:34.190 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.ReserveCommit (3026 ms) 2023-09-22T13:22:34.190 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.ReserveAbort 2023-09-22T13:22:37.210 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.ReserveAbort (3020 ms) 2023-09-22T13:22:37.210 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.ManualCleanup 2023-09-22T13:22:40.235 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.ManualCleanup (3025 ms) 2023-09-22T13:22:40.235 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.Cleanup 2023-09-22T13:22:43.254 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.Cleanup (3019 ms) 2023-09-22T13:22:43.254 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.MultiProducer 2023-09-22T13:22:54.715 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.MultiProducer (11462 ms) 2023-09-22T13:22:54.716 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.AsyncConsumer 2023-09-22T13:22:57.729 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.AsyncConsumer (3014 ms) 2023-09-22T13:22:57.730 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.MultiProducerConsumer 2023-09-22T13:23:10.782 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.MultiProducerConsumer (13053 ms) 2023-09-22T13:23:10.783 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.ReserveSpillover 2023-09-22T13:23:14.803 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.ReserveSpillover (4020 ms) 2023-09-22T13:23:14.803 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.CommitSpillover 2023-09-22T13:23:19.838 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.CommitSpillover (5035 ms) 2023-09-22T13:23:19.838 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestCls2PCQueue.AbortSpillover 2023-09-22T13:23:24.885 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestCls2PCQueue.AbortSpillover (5048 ms) 2023-09-22T13:23:24.886 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 22 tests from TestCls2PCQueue (89837 ms total) 2023-09-22T13:23:24.886 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:23:24.886 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:23:24.886 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 22 tests from 1 test suite ran. (89837 ms total) 2023-09-22T13:23:24.886 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 22 tests. 2023-09-22T13:23:24.888 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-09-22T13:23:24.888 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-22T13:23:24.961 INFO:tasks.workunit:Running workunits matching rgw/test_rgw_gc_log.sh on client.0... 2023-09-22T13:23:24.962 INFO:tasks.workunit:Running workunit rgw/test_rgw_gc_log.sh... 2023-09-22T13:23:24.962 DEBUG:teuthology.orchestra.run.smithi096:workunit test rgw/test_rgw_gc_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=31c126b7c1c0c06b66d9d19c5bfc51c46b503392 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_gc_log.sh 2023-09-22T13:23:25.046 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:23:25.046 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:23:25.046 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 6 tests from 1 test suite. 2023-09-22T13:23:25.047 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:23:26.908 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 6 tests from rgw_gc_log 2023-09-22T13:23:26.908 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] rgw_gc_log.init_existing_queue 2023-09-22T13:23:26.944 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] rgw_gc_log.init_existing_queue (37 ms) 2023-09-22T13:23:26.944 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] rgw_gc_log.init_existing_omap 2023-09-22T13:23:26.956 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] rgw_gc_log.init_existing_omap (12 ms) 2023-09-22T13:23:26.956 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] rgw_gc_log.enqueue1_after_init 2023-09-22T13:23:26.958 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] rgw_gc_log.enqueue1_after_init (2 ms) 2023-09-22T13:23:26.958 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] rgw_gc_log.enqueue2_before_init 2023-09-22T13:23:26.960 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] rgw_gc_log.enqueue2_before_init (2 ms) 2023-09-22T13:23:26.960 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] rgw_gc_log.defer1_after_init 2023-09-22T13:23:26.962 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] rgw_gc_log.defer1_after_init (2 ms) 2023-09-22T13:23:26.962 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] rgw_gc_log.defer2_before_init 2023-09-22T13:23:26.963 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] rgw_gc_log.defer2_before_init (1 ms) 2023-09-22T13:23:26.963 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 6 tests from rgw_gc_log (56 ms total) 2023-09-22T13:23:26.963 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:23:26.963 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:23:27.895 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 6 tests from 1 test suite ran. (2848 ms total) 2023-09-22T13:23:27.895 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 6 tests. 2023-09-22T13:23:27.899 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-09-22T13:23:27.899 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-22T13:23:27.929 INFO:tasks.workunit:Running workunits matching rgw/test_rgw_obj.sh on client.0... 2023-09-22T13:23:27.930 INFO:tasks.workunit:Running workunit rgw/test_rgw_obj.sh... 2023-09-22T13:23:27.931 DEBUG:teuthology.orchestra.run.smithi096:workunit test rgw/test_rgw_obj.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=31c126b7c1c0c06b66d9d19c5bfc51c46b503392 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_obj.sh 2023-09-22T13:23:28.011 INFO:tasks.workunit.client.0.smithi096.stdout:Running main() from gmock_main.cc 2023-09-22T13:23:28.011 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 4 tests from 1 test suite. 2023-09-22T13:23:28.011 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:23:28.011 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 4 tests from TestRGWObj 2023-09-22T13:23:28.011 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestRGWObj.underscore 2023-09-22T13:23:28.011 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj 2023-09-22T13:23:28.011 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj 2023-09-22T13:23:28.011 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: _obj ns= i= 2023-09-22T13:23:28.012 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj 2023-09-22T13:23:28.012 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj 2023-09-22T13:23:28.012 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: _obj ns= i= 2023-09-22T13:23:28.012 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj 2023-09-22T13:23:28.012 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj 2023-09-22T13:23:28.012 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: _obj ns= i= 2023-09-22T13:23:28.012 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.012 INFO:tasks.workunit.client.0.smithi096.stdout: "obj1": { 2023-09-22T13:23:28.012 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.012 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.012 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.012 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.012 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.013 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.013 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.013 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.013 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.013 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.013 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.013 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.013 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_obj", 2023-09-22T13:23:28.013 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.013 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.013 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.013 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.014 INFO:tasks.workunit.client.0.smithi096.stdout: "obj2": { 2023-09-22T13:23:28.014 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.014 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.014 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.014 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.014 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.014 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.014 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.014 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.014 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.014 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.015 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.015 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.015 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_obj", 2023-09-22T13:23:28.015 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.015 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.015 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.015 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.015 INFO:tasks.workunit.client.0.smithi096.stdout: "obj3": { 2023-09-22T13:23:28.015 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.015 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.015 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.015 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.016 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.016 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.016 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.016 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.016 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.016 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.016 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.016 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.016 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_obj", 2023-09-22T13:23:28.016 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.016 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.017 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.017 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.017 INFO:tasks.workunit.client.0.smithi096.stdout: "obj3-2": { 2023-09-22T13:23:28.017 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.017 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.017 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.017 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.017 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.017 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.017 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.017 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.017 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.017 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.018 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.018 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.018 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_obj", 2023-09-22T13:23:28.018 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.018 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.018 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.018 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.018 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.018 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj 2023-09-22T13:23:28.018 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj 2023-09-22T13:23:28.018 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: _obj ns=ns i= 2023-09-22T13:23:28.018 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj 2023-09-22T13:23:28.019 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj 2023-09-22T13:23:28.019 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: _obj ns=ns i= 2023-09-22T13:23:28.019 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj 2023-09-22T13:23:28.019 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj 2023-09-22T13:23:28.019 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: _obj ns=ns i= 2023-09-22T13:23:28.019 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.019 INFO:tasks.workunit.client.0.smithi096.stdout: "obj1": { 2023-09-22T13:23:28.019 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.019 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.019 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.019 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.019 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.020 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.020 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.020 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.020 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.020 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.020 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.020 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.020 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_obj", 2023-09-22T13:23:28.020 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.020 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.020 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.020 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.020 INFO:tasks.workunit.client.0.smithi096.stdout: "obj2": { 2023-09-22T13:23:28.021 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.021 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.021 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.021 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.021 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.021 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.021 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.021 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.021 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.021 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.021 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.021 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.022 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_obj", 2023-09-22T13:23:28.022 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.022 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.022 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.022 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.022 INFO:tasks.workunit.client.0.smithi096.stdout: "obj3": { 2023-09-22T13:23:28.022 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.022 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.022 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.022 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.022 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.023 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.023 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.023 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.023 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.023 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.023 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.023 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.023 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_obj", 2023-09-22T13:23:28.023 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.023 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.023 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.023 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.024 INFO:tasks.workunit.client.0.smithi096.stdout: "obj3-2": { 2023-09-22T13:23:28.024 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.024 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.024 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.024 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.024 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.024 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.024 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.024 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.024 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.024 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.024 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.025 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.025 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_obj", 2023-09-22T13:23:28.025 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.025 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.025 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.025 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.025 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.025 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj[v1] 2023-09-22T13:23:28.025 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj[v1] 2023-09-22T13:23:28.025 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: _obj ns= i=v1 2023-09-22T13:23:28.025 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj[v1] 2023-09-22T13:23:28.025 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj[v1] 2023-09-22T13:23:28.025 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: _obj ns= i=v1 2023-09-22T13:23:28.026 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj[v1] 2023-09-22T13:23:28.026 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj[v1] 2023-09-22T13:23:28.026 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: _obj ns= i=v1 2023-09-22T13:23:28.026 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.026 INFO:tasks.workunit.client.0.smithi096.stdout: "obj1": { 2023-09-22T13:23:28.026 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.026 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.026 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.026 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.026 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.026 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.026 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.027 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.027 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.027 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.027 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.027 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.027 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_obj", 2023-09-22T13:23:28.027 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "v1", 2023-09-22T13:23:28.027 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.027 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.027 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.027 INFO:tasks.workunit.client.0.smithi096.stdout: "obj2": { 2023-09-22T13:23:28.027 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.028 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.028 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.028 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.028 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.028 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.028 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.028 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.028 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.028 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.028 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.028 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.028 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_obj", 2023-09-22T13:23:28.028 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "v1", 2023-09-22T13:23:28.029 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.029 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.029 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.029 INFO:tasks.workunit.client.0.smithi096.stdout: "obj3": { 2023-09-22T13:23:28.029 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.029 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.029 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.029 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.029 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.029 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.029 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.029 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.029 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.030 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.030 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.030 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.030 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_obj", 2023-09-22T13:23:28.030 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.030 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.030 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.030 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.030 INFO:tasks.workunit.client.0.smithi096.stdout: "obj3-2": { 2023-09-22T13:23:28.030 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.030 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.030 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.031 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.031 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.031 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.031 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.031 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.031 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.031 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.031 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.031 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.031 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_obj", 2023-09-22T13:23:28.031 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "v1", 2023-09-22T13:23:28.031 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.031 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.032 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.032 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.032 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj[v1] 2023-09-22T13:23:28.032 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj[v1] 2023-09-22T13:23:28.032 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: _obj ns=ns i=v1 2023-09-22T13:23:28.032 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj[v1] 2023-09-22T13:23:28.032 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj[v1] 2023-09-22T13:23:28.032 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: _obj ns=ns i=v1 2023-09-22T13:23:28.032 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj[v1] 2023-09-22T13:23:28.032 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: _obj[v1] 2023-09-22T13:23:28.032 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: _obj ns=ns i=v1 2023-09-22T13:23:28.032 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.033 INFO:tasks.workunit.client.0.smithi096.stdout: "obj1": { 2023-09-22T13:23:28.033 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.033 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.033 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.033 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.033 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.033 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.033 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.033 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.033 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.033 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.033 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.034 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.034 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_obj", 2023-09-22T13:23:28.034 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "v1", 2023-09-22T13:23:28.034 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.034 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.034 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.034 INFO:tasks.workunit.client.0.smithi096.stdout: "obj2": { 2023-09-22T13:23:28.034 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.034 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.034 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.034 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.034 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.035 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.035 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.035 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.035 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.035 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.035 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.035 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.035 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_obj", 2023-09-22T13:23:28.035 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "v1", 2023-09-22T13:23:28.035 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.035 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.035 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.035 INFO:tasks.workunit.client.0.smithi096.stdout: "obj3": { 2023-09-22T13:23:28.036 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.036 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.036 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.036 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.036 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.036 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.036 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.036 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.036 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.036 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.036 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.036 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.036 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_obj", 2023-09-22T13:23:28.037 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.037 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.037 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.037 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.037 INFO:tasks.workunit.client.0.smithi096.stdout: "obj3-2": { 2023-09-22T13:23:28.037 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.037 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.037 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.037 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.037 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.037 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.037 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.037 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.038 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.038 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.038 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.038 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.038 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_obj", 2023-09-22T13:23:28.038 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "v1", 2023-09-22T13:23:28.038 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.038 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.038 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.038 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.038 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestRGWObj.underscore (0 ms) 2023-09-22T13:23:28.038 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestRGWObj.no_underscore 2023-09-22T13:23:28.039 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj 2023-09-22T13:23:28.039 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj 2023-09-22T13:23:28.039 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: obj ns= i= 2023-09-22T13:23:28.039 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj 2023-09-22T13:23:28.039 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj 2023-09-22T13:23:28.039 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: obj ns= i= 2023-09-22T13:23:28.039 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj 2023-09-22T13:23:28.039 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj 2023-09-22T13:23:28.039 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: obj ns= i= 2023-09-22T13:23:28.039 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.039 INFO:tasks.workunit.client.0.smithi096.stdout: "obj1": { 2023-09-22T13:23:28.039 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.039 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.040 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.040 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.040 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.040 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.040 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.040 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.040 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.040 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.040 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.040 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.040 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "obj", 2023-09-22T13:23:28.041 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.041 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.041 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.041 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.041 INFO:tasks.workunit.client.0.smithi096.stdout: "obj2": { 2023-09-22T13:23:28.041 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.041 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.041 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.041 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.041 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.041 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.041 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.042 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.042 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.042 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.042 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.042 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.042 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "obj", 2023-09-22T13:23:28.042 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.042 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.042 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.042 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.042 INFO:tasks.workunit.client.0.smithi096.stdout: "obj3": { 2023-09-22T13:23:28.042 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.043 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.043 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.043 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.043 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.043 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.043 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.043 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.043 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.043 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.043 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.043 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.043 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "obj", 2023-09-22T13:23:28.044 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.044 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.044 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.044 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.044 INFO:tasks.workunit.client.0.smithi096.stdout: "obj3-2": { 2023-09-22T13:23:28.045 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.045 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.046 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.046 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.046 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.046 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.046 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.046 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.046 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.046 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.046 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.046 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.046 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "obj", 2023-09-22T13:23:28.046 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.047 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.047 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.047 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.047 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.047 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj 2023-09-22T13:23:28.047 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj 2023-09-22T13:23:28.047 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: obj ns=ns i= 2023-09-22T13:23:28.047 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj 2023-09-22T13:23:28.047 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj 2023-09-22T13:23:28.047 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: obj ns=ns i= 2023-09-22T13:23:28.047 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj 2023-09-22T13:23:28.047 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj 2023-09-22T13:23:28.048 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: obj ns=ns i= 2023-09-22T13:23:28.048 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.048 INFO:tasks.workunit.client.0.smithi096.stdout: "obj1": { 2023-09-22T13:23:28.048 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.048 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.048 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.048 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.048 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.048 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.048 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.048 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.049 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.049 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.049 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.049 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.049 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "obj", 2023-09-22T13:23:28.049 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.049 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.049 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.049 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.049 INFO:tasks.workunit.client.0.smithi096.stdout: "obj2": { 2023-09-22T13:23:28.049 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.049 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.050 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.050 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.050 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.050 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.050 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.050 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.050 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.050 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.050 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.050 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.050 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "obj", 2023-09-22T13:23:28.051 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.051 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.051 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.051 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.051 INFO:tasks.workunit.client.0.smithi096.stdout: "obj3": { 2023-09-22T13:23:28.051 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.051 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.051 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.051 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.051 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.051 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.051 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.052 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.052 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.052 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.052 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.052 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.052 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "obj", 2023-09-22T13:23:28.052 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.052 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.052 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.052 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.052 INFO:tasks.workunit.client.0.smithi096.stdout: "obj3-2": { 2023-09-22T13:23:28.052 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.053 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.053 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.053 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.053 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.053 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.053 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.053 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.053 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.053 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.053 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.053 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.053 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "obj", 2023-09-22T13:23:28.054 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.054 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.054 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.054 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.054 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.054 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj[v1] 2023-09-22T13:23:28.054 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj[v1] 2023-09-22T13:23:28.054 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: obj ns= i=v1 2023-09-22T13:23:28.054 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj[v1] 2023-09-22T13:23:28.054 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj[v1] 2023-09-22T13:23:28.054 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: obj ns= i=v1 2023-09-22T13:23:28.054 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj[v1] 2023-09-22T13:23:28.055 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj[v1] 2023-09-22T13:23:28.055 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: obj ns= i=v1 2023-09-22T13:23:28.055 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.055 INFO:tasks.workunit.client.0.smithi096.stdout: "obj1": { 2023-09-22T13:23:28.055 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.055 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.055 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.055 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.055 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.055 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.055 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.055 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.055 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.056 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.056 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.056 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.056 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "obj", 2023-09-22T13:23:28.056 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "v1", 2023-09-22T13:23:28.056 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.056 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.056 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.056 INFO:tasks.workunit.client.0.smithi096.stdout: "obj2": { 2023-09-22T13:23:28.056 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.056 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.056 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.057 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.057 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.057 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.057 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.057 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.057 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.057 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.057 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.057 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.057 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "obj", 2023-09-22T13:23:28.057 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "v1", 2023-09-22T13:23:28.057 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.057 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.058 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.058 INFO:tasks.workunit.client.0.smithi096.stdout: "obj3": { 2023-09-22T13:23:28.058 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.058 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.058 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.058 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.058 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.058 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.058 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.058 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.058 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.059 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.059 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.059 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.059 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "obj", 2023-09-22T13:23:28.059 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.059 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.059 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.059 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.059 INFO:tasks.workunit.client.0.smithi096.stdout: "obj3-2": { 2023-09-22T13:23:28.059 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.059 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.059 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.060 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.060 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.060 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.060 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.060 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.060 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.060 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.060 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.060 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.060 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "obj", 2023-09-22T13:23:28.060 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "v1", 2023-09-22T13:23:28.060 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.061 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.061 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.061 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.061 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj[v1] 2023-09-22T13:23:28.061 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj[v1] 2023-09-22T13:23:28.061 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: obj ns=ns i=v1 2023-09-22T13:23:28.061 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj[v1] 2023-09-22T13:23:28.061 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj[v1] 2023-09-22T13:23:28.061 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: obj ns=ns i=v1 2023-09-22T13:23:28.061 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj[v1] 2023-09-22T13:23:28.061 INFO:tasks.workunit.client.0.smithi096.stdout:parsed: obj[v1] 2023-09-22T13:23:28.061 INFO:tasks.workunit.client.0.smithi096.stdout:stripped: obj ns=ns i=v1 2023-09-22T13:23:28.062 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.062 INFO:tasks.workunit.client.0.smithi096.stdout: "obj1": { 2023-09-22T13:23:28.062 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.062 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.062 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.062 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.062 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.062 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.062 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.062 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.062 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.062 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.063 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.063 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.063 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "obj", 2023-09-22T13:23:28.063 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "v1", 2023-09-22T13:23:28.063 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.063 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.063 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.063 INFO:tasks.workunit.client.0.smithi096.stdout: "obj2": { 2023-09-22T13:23:28.063 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.063 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.063 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.063 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.064 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.064 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.064 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.064 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.064 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.064 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.064 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.064 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.064 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "obj", 2023-09-22T13:23:28.064 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "v1", 2023-09-22T13:23:28.064 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.064 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.065 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.065 INFO:tasks.workunit.client.0.smithi096.stdout: "obj3": { 2023-09-22T13:23:28.065 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.065 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.065 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.065 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.065 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.065 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.065 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.065 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.065 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.065 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.066 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.066 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.066 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "obj", 2023-09-22T13:23:28.066 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.066 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.066 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.066 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.066 INFO:tasks.workunit.client.0.smithi096.stdout: "obj3-2": { 2023-09-22T13:23:28.066 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.066 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.066 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.066 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.067 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.067 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.067 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.067 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.067 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.067 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.067 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.067 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.067 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "obj", 2023-09-22T13:23:28.067 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "v1", 2023-09-22T13:23:28.067 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.067 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.068 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.068 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.068 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestRGWObj.no_underscore (0 ms) 2023-09-22T13:23:28.068 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestRGWObj.obj_to_raw 2023-09-22T13:23:28.068 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.068 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.068 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.068 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.068 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.068 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.068 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.068 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.068 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.069 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.069 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.069 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.069 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.069 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.069 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.069 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.069 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.069 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.069 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.069 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.069 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.070 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.070 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.070 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.070 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.070 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.070 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.070 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.070 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.070 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.070 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.070 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.070 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.071 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.071 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.071 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": "default-placement.data", 2023-09-22T13:23:28.071 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker_myobj", 2023-09-22T13:23:28.071 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.071 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.071 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.071 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.071 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.071 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.071 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.071 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.072 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.072 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.072 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.072 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.072 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.072 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.072 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.072 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.072 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.072 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.072 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.072 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.073 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.073 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.073 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.073 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.073 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.073 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.073 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.073 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.073 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.073 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.073 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.074 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.074 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.074 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.074 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.074 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.074 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.074 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.074 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.074 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.074 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.074 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.074 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.075 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.075 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.075 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.075 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.075 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.075 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.075 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.075 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.075 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.075 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.075 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.075 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.076 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.076 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.076 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.076 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.076 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker_myobj", 2023-09-22T13:23:28.076 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.076 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.076 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.076 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.076 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.076 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.076 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.076 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.077 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.077 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.077 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.077 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.077 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.077 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.077 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.077 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.077 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.077 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.077 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.078 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.078 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.078 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.078 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.078 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.078 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.078 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.078 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.078 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.078 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.078 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.078 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.079 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.079 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.079 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.079 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.079 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.079 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.079 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.079 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.079 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.079 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.079 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.079 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.079 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.080 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.080 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.080 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.080 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.080 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.080 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.080 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.080 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.080 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.080 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.080 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.080 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.080 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.081 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.081 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": "default-placement.data", 2023-09-22T13:23:28.081 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns_myobj", 2023-09-22T13:23:28.081 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.081 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.081 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.081 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.081 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.081 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.081 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.081 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.081 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.082 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.082 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.082 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.082 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.082 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.082 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.082 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.082 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.082 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.082 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.082 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.082 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.083 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.083 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.083 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.083 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.083 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.083 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.083 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.083 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.083 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.083 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.083 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.083 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.083 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.084 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.084 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.084 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.084 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.084 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.084 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.084 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.084 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.084 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.084 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.084 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.084 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.085 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.085 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.085 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.085 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.085 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.085 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.085 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.085 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.085 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.085 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.085 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.085 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.085 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.086 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns_myobj", 2023-09-22T13:23:28.086 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.086 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.086 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.086 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.086 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.086 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.086 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.086 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.086 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.086 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.086 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.086 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.087 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.087 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.087 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.087 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.087 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.087 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.087 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.087 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.087 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.087 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.087 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.087 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.087 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.088 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.088 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.088 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.088 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.088 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.088 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.088 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.088 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.088 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.088 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.088 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.088 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.089 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.089 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.089 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.089 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.089 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.090 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.090 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.090 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.090 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.090 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.090 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.090 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.090 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.090 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.090 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.091 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.091 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.091 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.091 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.091 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.091 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.091 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": "default-placement.data", 2023-09-22T13:23:28.091 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__:inst_myobj", 2023-09-22T13:23:28.091 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.091 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.091 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.091 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.091 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.092 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.092 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.092 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.092 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.092 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.092 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.092 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.092 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.092 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.092 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.092 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.092 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.093 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.093 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.093 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.093 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.093 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.093 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.093 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.093 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.093 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.093 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.093 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.093 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.093 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.094 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.094 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.094 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.094 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.094 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.094 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.094 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.094 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.094 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.094 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.094 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.094 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.095 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.095 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.095 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.095 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.095 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.095 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.095 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.095 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.096 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.096 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.096 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.096 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.096 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.096 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.096 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.096 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.096 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.096 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__:inst_myobj", 2023-09-22T13:23:28.096 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.096 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.097 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.097 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.097 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.097 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.097 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.097 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.097 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.097 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.097 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.097 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.097 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.097 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.097 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.098 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.098 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.098 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.098 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.098 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.098 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.098 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.098 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.098 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.098 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.098 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.099 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.099 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.099 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.099 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.099 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.099 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.099 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.099 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.099 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.099 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.099 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.099 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.099 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.100 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.100 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.100 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.100 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.100 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.100 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.100 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.100 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.100 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.100 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.100 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.100 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.100 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.101 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.101 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.101 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.101 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.101 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.101 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.101 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": "default-placement.data", 2023-09-22T13:23:28.101 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns:inst_myobj", 2023-09-22T13:23:28.101 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.101 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.101 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.101 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.101 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.102 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.102 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.102 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.102 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.102 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.102 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.102 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.102 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.102 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.102 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.102 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.102 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.102 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.103 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.103 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.103 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.103 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.103 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.103 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.103 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.103 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.103 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.103 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.103 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.103 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.104 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.104 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.104 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.104 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.104 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.104 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.104 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.104 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.104 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.104 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.104 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.104 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.104 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.105 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.105 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.105 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.105 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.105 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.105 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.105 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.105 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.105 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.105 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.105 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.105 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.105 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.106 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.106 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.106 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.106 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns:inst_myobj", 2023-09-22T13:23:28.106 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.106 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.106 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.106 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.106 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.107 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.107 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.107 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.107 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.107 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.107 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.108 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.108 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.108 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.108 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.108 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.108 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.108 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.108 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.108 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.108 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.108 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.108 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.108 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.109 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.109 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.109 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.109 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.109 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.109 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.109 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.109 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.109 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.109 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.109 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.109 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.109 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.110 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.110 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.110 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.110 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.110 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.110 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.110 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.110 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.110 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.110 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.110 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.110 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.111 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.111 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.111 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.111 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.111 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.111 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.111 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.111 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.111 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.111 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": "default-placement.data", 2023-09-22T13:23:28.111 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker___myobj", 2023-09-22T13:23:28.111 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "marker__myobj" 2023-09-22T13:23:28.111 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.112 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.112 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.112 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.112 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.112 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.112 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.112 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.112 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.112 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.112 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.112 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.112 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.113 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.113 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.113 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.113 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.113 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.113 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.113 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.113 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.113 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.113 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.113 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.113 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.113 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.114 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.114 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.114 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.114 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.114 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.114 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.114 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.114 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.114 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.114 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.114 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.114 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.115 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.115 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.115 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.115 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.115 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.115 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.115 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.115 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.115 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.115 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.116 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.116 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.116 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.116 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.116 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.116 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.116 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.116 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.116 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.116 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.116 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker___myobj", 2023-09-22T13:23:28.116 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "marker__myobj" 2023-09-22T13:23:28.117 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.117 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.117 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.117 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.117 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.117 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.117 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.117 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.118 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.118 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.118 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.118 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.118 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.118 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.118 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.118 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.118 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.118 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.118 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.118 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.119 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.119 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.119 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.119 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.119 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.119 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.119 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.119 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.119 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.119 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.119 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.119 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.119 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.120 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.120 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.120 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.120 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.120 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.120 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.120 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.120 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.120 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.120 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.120 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.120 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.120 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.121 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.121 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.121 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.121 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.121 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.121 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.121 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.121 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.121 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.121 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.121 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.121 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": "default-placement.data", 2023-09-22T13:23:28.121 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns__myobj", 2023-09-22T13:23:28.122 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.122 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.122 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.122 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.122 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.122 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.122 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.122 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.122 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.122 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.122 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.122 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.123 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.123 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.123 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.123 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.123 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.123 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.123 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.123 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.123 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.123 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.123 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.123 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.123 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.124 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.124 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.124 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.124 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.124 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.124 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.124 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.124 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.124 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.124 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.124 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.124 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.124 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.125 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.125 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.125 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.125 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.125 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.125 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.125 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.125 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.125 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.125 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.125 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.125 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.126 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.126 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.126 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.126 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.126 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.126 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.126 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.126 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.126 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.126 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns__myobj", 2023-09-22T13:23:28.126 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.126 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.126 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.127 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.127 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.127 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.127 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.127 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.127 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.127 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.127 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.127 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.127 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.127 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.127 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.128 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.128 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.128 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.128 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.128 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.128 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.128 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.128 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.128 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.128 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.128 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.128 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.128 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.129 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.129 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.129 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.129 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.129 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.129 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.129 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.129 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.129 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.129 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.129 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.129 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.130 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.130 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.130 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.130 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.130 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.130 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.130 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.130 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.130 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.130 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.130 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.130 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.130 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.131 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.131 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.131 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.131 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.131 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.131 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": "default-placement.data", 2023-09-22T13:23:28.131 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__:inst__myobj", 2023-09-22T13:23:28.131 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "marker__myobj" 2023-09-22T13:23:28.131 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.131 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.131 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.131 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.132 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.132 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.132 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.132 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.132 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.132 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.132 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.132 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.132 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.132 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.132 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.132 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.132 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.132 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.133 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.133 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.133 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.133 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.133 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.133 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.133 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.133 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.133 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.133 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.133 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.134 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.134 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.134 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.134 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.134 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.134 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.134 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.134 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.134 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.134 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.134 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.134 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.135 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.135 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.135 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.135 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.135 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.135 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.135 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.135 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.135 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.135 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.135 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.135 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.136 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.136 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.136 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.136 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.136 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.136 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__:inst__myobj", 2023-09-22T13:23:28.136 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "marker__myobj" 2023-09-22T13:23:28.136 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.136 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.136 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.136 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.136 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.136 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.137 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.137 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.137 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.137 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.137 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.137 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.137 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.137 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.137 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.137 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.137 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.137 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.137 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.138 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.138 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.138 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.138 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.138 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.138 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.138 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.138 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.138 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.138 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.138 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.138 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.139 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.139 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.139 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.139 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.139 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.139 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.139 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.139 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.139 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.139 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.139 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.139 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.139 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.140 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.140 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.140 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.140 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.140 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.140 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.140 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.140 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.140 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.140 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.140 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.140 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.140 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.141 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": "default-placement.data", 2023-09-22T13:23:28.141 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns:inst__myobj", 2023-09-22T13:23:28.141 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.141 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.141 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.141 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.141 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.141 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.141 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.141 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.141 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.141 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.141 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.142 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.142 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.142 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.142 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.142 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.142 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.142 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.142 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.142 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.142 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.142 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.143 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.143 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.143 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.143 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.143 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.143 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.143 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.143 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.143 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.143 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.143 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.143 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.143 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.144 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.144 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.144 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.144 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.144 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.144 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.144 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.144 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.144 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.144 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.145 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.145 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.145 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.145 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.145 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.145 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.145 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.145 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.145 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.145 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.145 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.145 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.146 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.146 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.146 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns:inst__myobj", 2023-09-22T13:23:28.146 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.146 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.146 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.146 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.146 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.146 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.146 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.146 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.146 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.146 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.147 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.147 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.147 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.147 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.147 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.147 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.147 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.147 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.147 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.147 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.147 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.147 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.147 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.148 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.148 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.148 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.148 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.148 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.148 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.148 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.148 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.148 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.148 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.148 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.148 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.149 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.149 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.149 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.149 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.149 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.149 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.149 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.149 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.149 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.149 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.149 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.149 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.149 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.150 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.150 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.150 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.150 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.150 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.150 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.150 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.150 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.150 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.150 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.150 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": "default-placement.data", 2023-09-22T13:23:28.150 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker___myobj_", 2023-09-22T13:23:28.151 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "marker__myobj_" 2023-09-22T13:23:28.151 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.151 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.151 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.151 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.151 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.151 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.151 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.151 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.151 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.151 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.151 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.151 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.152 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.152 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.152 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.152 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.152 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.152 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.152 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.152 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.152 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.152 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.152 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.152 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.152 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.153 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.153 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.153 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.153 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.153 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.153 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.153 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.153 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.153 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.153 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.153 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.153 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.153 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.154 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.154 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.154 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.154 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.154 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.154 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.154 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.154 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.154 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.154 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.154 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.154 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.154 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.155 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.155 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.155 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.155 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.155 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.155 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.155 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.155 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker___myobj_", 2023-09-22T13:23:28.155 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "marker__myobj_" 2023-09-22T13:23:28.155 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.155 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.155 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.156 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.156 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.156 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.156 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.156 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.156 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.156 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.156 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.156 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.156 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.156 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.156 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.157 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.157 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.157 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.157 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.157 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.157 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.157 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.157 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.157 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.157 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.157 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.157 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.157 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.158 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.158 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.158 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.158 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.158 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.158 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.158 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.158 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.158 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.158 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.158 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.158 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.158 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.159 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.159 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.159 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.159 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.159 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.159 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.159 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.159 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.159 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.159 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.159 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.159 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.159 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.160 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.160 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.160 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.160 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": "default-placement.data", 2023-09-22T13:23:28.160 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns__myobj_", 2023-09-22T13:23:28.160 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.160 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.160 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.160 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.160 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.160 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.160 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.160 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.161 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.161 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.161 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.161 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.161 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.161 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.161 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.161 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.161 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.161 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.161 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.161 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.162 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.162 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.162 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.162 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.162 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.162 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.162 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.162 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.162 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.162 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.162 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.162 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.163 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.163 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.163 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.163 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.163 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.163 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.163 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.163 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.163 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.163 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.163 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.163 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.164 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.164 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.164 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.164 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.164 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.164 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.164 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.164 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.164 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.164 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.164 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.164 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.164 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.164 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.165 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.165 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns__myobj_", 2023-09-22T13:23:28.165 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.165 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.165 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.165 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.165 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.165 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.165 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.165 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.165 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.165 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.165 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.166 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.166 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.166 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.166 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.166 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.166 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.166 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.166 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.166 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.166 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.166 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.166 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.167 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.167 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.167 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.167 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.167 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.167 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.167 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.167 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.167 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.167 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.167 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.167 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.167 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.168 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.168 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.168 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.168 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.168 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.168 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.168 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.168 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.168 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.168 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.168 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.168 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.168 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.169 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.169 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.169 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.169 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.169 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.169 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.169 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.169 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.169 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.169 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": "default-placement.data", 2023-09-22T13:23:28.169 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__:inst__myobj_", 2023-09-22T13:23:28.169 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "marker__myobj_" 2023-09-22T13:23:28.169 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.170 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.170 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.170 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.170 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.170 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.170 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.170 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.170 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.170 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.170 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.170 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.170 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.170 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.171 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.171 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.171 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.171 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.171 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.171 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.171 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.171 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.171 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.171 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.171 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.171 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.172 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.172 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.172 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.172 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.172 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.172 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.172 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.172 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.172 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.172 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.172 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.172 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.172 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.173 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.173 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.173 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.173 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.173 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.173 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.173 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.173 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.173 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.173 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.173 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.173 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.173 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.174 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.174 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.174 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.174 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.174 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.174 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.174 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__:inst__myobj_", 2023-09-22T13:23:28.174 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "marker__myobj_" 2023-09-22T13:23:28.174 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.174 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.174 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.174 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.174 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.175 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.175 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.175 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.175 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.175 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.175 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.175 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.175 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.175 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.175 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.175 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.175 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.176 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.176 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.176 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.176 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.176 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.176 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.176 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.176 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.176 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.176 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.176 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.176 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.177 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.177 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.177 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.177 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.177 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.177 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.177 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.177 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.177 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.177 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.177 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.177 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.177 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.178 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.178 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.178 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.178 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.178 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.178 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.178 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.178 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.178 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.178 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.178 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.178 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.178 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.179 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.179 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.179 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": "default-placement.data", 2023-09-22T13:23:28.179 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns:inst__myobj_", 2023-09-22T13:23:28.179 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.179 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.179 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.179 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.179 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.179 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.179 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "test", 2023-09-22T13:23:28.179 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.179 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.180 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.180 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.180 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": "", 2023-09-22T13:23:28.180 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.180 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": "" 2023-09-22T13:23:28.180 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.180 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.180 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.180 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.180 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.180 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.180 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.180 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.181 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.181 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.181 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.181 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.181 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.181 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.181 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.181 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.181 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.181 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.181 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.181 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.181 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.182 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.182 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.182 INFO:tasks.workunit.client.0.smithi096.stdout: "obj": { 2023-09-22T13:23:28.182 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.182 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.182 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.182 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.182 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.182 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.182 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.182 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.182 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.183 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.183 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.183 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.183 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.183 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.183 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.183 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.183 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.183 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.183 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.183 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.183 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.183 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns:inst__myobj_", 2023-09-22T13:23:28.184 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.184 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.184 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.184 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.184 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.184 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.184 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.184 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.184 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.184 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.184 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.184 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.184 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.185 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.185 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.185 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.185 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.185 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.185 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.185 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.185 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.185 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.185 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.185 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestRGWObj.obj_to_raw (1 ms) 2023-09-22T13:23:28.185 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] TestRGWObj.old_to_raw 2023-09-22T13:23:28.185 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.186 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.186 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.186 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker_myobj", 2023-09-22T13:23:28.186 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.186 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.186 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.186 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.186 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.186 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.186 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.186 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.186 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.186 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.187 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.187 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.187 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.187 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.187 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.187 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.187 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.187 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.187 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.187 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.187 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.187 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.188 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.188 INFO:tasks.workunit.client.0.smithi096.stdout:raw=.data-pool:marker_myobj 2023-09-22T13:23:28.188 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.188 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj2": { 2023-09-22T13:23:28.188 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.188 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker_myobj", 2023-09-22T13:23:28.188 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.188 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.188 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.188 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.188 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj2": { 2023-09-22T13:23:28.188 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.188 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.188 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.189 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.189 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.189 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.189 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.189 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.189 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.189 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.189 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.189 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.189 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.189 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.189 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.189 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.190 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.190 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.190 INFO:tasks.workunit.client.0.smithi096.stdout:raw2=.data-pool:marker_myobj 2023-09-22T13:23:28.190 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.190 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.190 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.190 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns_myobj", 2023-09-22T13:23:28.190 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.190 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.190 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.190 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.190 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.191 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.191 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.191 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.191 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.191 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.191 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.191 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.191 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.191 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.191 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.191 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.191 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.191 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.192 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.192 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.192 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.192 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.192 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.192 INFO:tasks.workunit.client.0.smithi096.stdout:raw=.data-pool:marker__ns_myobj 2023-09-22T13:23:28.192 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.192 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj2": { 2023-09-22T13:23:28.192 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.192 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns_myobj", 2023-09-22T13:23:28.192 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.192 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.192 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.193 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.193 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj2": { 2023-09-22T13:23:28.193 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.193 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.193 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.193 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.193 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.193 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.193 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.193 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.193 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.193 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.194 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.194 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.194 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.194 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.194 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.194 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.194 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.194 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.194 INFO:tasks.workunit.client.0.smithi096.stdout:raw2=.data-pool:marker__ns_myobj 2023-09-22T13:23:28.194 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.194 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.194 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.194 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__:inst_myobj", 2023-09-22T13:23:28.195 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.195 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.195 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.195 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.195 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.195 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.195 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.195 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.195 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.195 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.195 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.195 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.196 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.196 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.196 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.196 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.196 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.196 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.196 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.196 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.196 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.196 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.196 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.196 INFO:tasks.workunit.client.0.smithi096.stdout:raw=.data-pool:marker__:inst_myobj 2023-09-22T13:23:28.197 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.197 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj2": { 2023-09-22T13:23:28.197 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.197 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__:inst_myobj", 2023-09-22T13:23:28.197 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.197 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.197 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.197 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.197 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj2": { 2023-09-22T13:23:28.197 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.197 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.197 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.197 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.198 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.198 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.198 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.198 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.198 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.198 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.198 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.198 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.198 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.198 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.198 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.198 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.199 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.199 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.199 INFO:tasks.workunit.client.0.smithi096.stdout:raw2=.data-pool:marker__:inst_myobj 2023-09-22T13:23:28.199 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.199 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.199 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.199 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns:inst_myobj", 2023-09-22T13:23:28.199 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.199 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.199 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.199 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.199 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.199 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.200 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.200 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.200 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.200 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.200 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.200 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.200 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.200 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.200 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.200 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.200 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.200 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.200 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.201 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.201 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.201 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.201 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.201 INFO:tasks.workunit.client.0.smithi096.stdout:raw=.data-pool:marker__ns:inst_myobj 2023-09-22T13:23:28.201 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.201 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj2": { 2023-09-22T13:23:28.201 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.201 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns:inst_myobj", 2023-09-22T13:23:28.201 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.201 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.201 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.201 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.202 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj2": { 2023-09-22T13:23:28.202 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.202 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.202 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.202 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.202 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.202 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.202 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.202 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.202 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.202 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.202 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.202 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.203 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "myobj", 2023-09-22T13:23:28.203 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.203 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.203 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.203 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.203 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.203 INFO:tasks.workunit.client.0.smithi096.stdout:raw2=.data-pool:marker__ns:inst_myobj 2023-09-22T13:23:28.203 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.203 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.203 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.203 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker___myobj", 2023-09-22T13:23:28.203 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "marker__myobj" 2023-09-22T13:23:28.204 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.204 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.204 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.204 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.204 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.204 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.204 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.204 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.204 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.204 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.204 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.204 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.204 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.205 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.205 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.205 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.205 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.205 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.205 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.205 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.205 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.205 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.205 INFO:tasks.workunit.client.0.smithi096.stdout:raw=.data-pool:marker___myobj 2023-09-22T13:23:28.205 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.205 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj2": { 2023-09-22T13:23:28.205 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.206 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker___myobj", 2023-09-22T13:23:28.206 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "marker__myobj" 2023-09-22T13:23:28.206 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.206 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.206 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.206 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj2": { 2023-09-22T13:23:28.206 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.206 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.206 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.206 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.206 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.206 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.207 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.207 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.207 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.207 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.207 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.207 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.207 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.207 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.207 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.207 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.207 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.207 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.208 INFO:tasks.workunit.client.0.smithi096.stdout:raw2=.data-pool:marker___myobj 2023-09-22T13:23:28.208 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.208 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.208 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.208 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns__myobj", 2023-09-22T13:23:28.208 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.208 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.208 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.208 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.208 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.208 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.208 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.208 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.209 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.209 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.209 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.209 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.209 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.209 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.209 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.209 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.209 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.209 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.209 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.209 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.210 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.210 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.210 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.210 INFO:tasks.workunit.client.0.smithi096.stdout:raw=.data-pool:marker__ns__myobj 2023-09-22T13:23:28.210 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.210 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj2": { 2023-09-22T13:23:28.210 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.210 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns__myobj", 2023-09-22T13:23:28.210 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.210 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.210 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.210 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.210 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj2": { 2023-09-22T13:23:28.211 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.211 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.211 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.211 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.211 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.211 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.211 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.211 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.211 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.211 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.211 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.211 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.211 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.212 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.212 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.212 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.212 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.212 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.212 INFO:tasks.workunit.client.0.smithi096.stdout:raw2=.data-pool:marker__ns__myobj 2023-09-22T13:23:28.212 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.212 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.212 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.212 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__:inst__myobj", 2023-09-22T13:23:28.212 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "marker__myobj" 2023-09-22T13:23:28.212 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.212 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.213 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.213 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.213 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.213 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.213 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.213 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.213 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.213 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.213 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.213 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.213 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.213 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.214 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.214 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.214 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.214 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.214 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.214 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.214 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.214 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.214 INFO:tasks.workunit.client.0.smithi096.stdout:raw=.data-pool:marker__:inst__myobj 2023-09-22T13:23:28.214 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.214 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj2": { 2023-09-22T13:23:28.214 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.215 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__:inst__myobj", 2023-09-22T13:23:28.215 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "marker__myobj" 2023-09-22T13:23:28.215 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.215 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.215 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.215 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj2": { 2023-09-22T13:23:28.215 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.215 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.215 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.215 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.215 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.215 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.215 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.216 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.216 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.216 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.216 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.216 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.216 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.216 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.216 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.216 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.216 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.216 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.216 INFO:tasks.workunit.client.0.smithi096.stdout:raw2=.data-pool:marker__:inst__myobj 2023-09-22T13:23:28.216 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.217 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.217 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.217 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns:inst__myobj", 2023-09-22T13:23:28.217 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.217 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.217 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.217 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.217 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.217 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.217 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.217 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.217 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.217 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.218 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.218 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.218 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.218 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.218 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.218 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.218 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.218 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.218 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.218 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.218 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.218 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.219 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.219 INFO:tasks.workunit.client.0.smithi096.stdout:raw=.data-pool:marker__ns:inst__myobj 2023-09-22T13:23:28.219 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.219 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj2": { 2023-09-22T13:23:28.219 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.219 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns:inst__myobj", 2023-09-22T13:23:28.219 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.219 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.219 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.219 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.219 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj2": { 2023-09-22T13:23:28.219 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.219 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.220 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.220 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.220 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.220 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.220 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.220 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.220 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.220 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.220 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.220 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.220 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj", 2023-09-22T13:23:28.220 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.221 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.221 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.221 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.221 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.221 INFO:tasks.workunit.client.0.smithi096.stdout:raw2=.data-pool:marker__ns:inst__myobj 2023-09-22T13:23:28.221 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.221 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.221 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.221 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker___myobj_", 2023-09-22T13:23:28.221 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "marker__myobj_" 2023-09-22T13:23:28.221 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.221 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.221 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.222 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.222 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.222 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.222 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.222 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.222 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.222 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.222 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.222 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.222 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.222 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.223 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.223 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.223 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.223 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.223 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.223 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.223 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.223 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.223 INFO:tasks.workunit.client.0.smithi096.stdout:raw=.data-pool:marker___myobj_ 2023-09-22T13:23:28.223 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.223 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj2": { 2023-09-22T13:23:28.223 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.223 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker___myobj_", 2023-09-22T13:23:28.224 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "marker__myobj_" 2023-09-22T13:23:28.224 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.224 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.224 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.224 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj2": { 2023-09-22T13:23:28.224 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.224 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.224 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.224 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.224 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.224 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.224 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.224 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.225 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.225 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.225 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.225 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.225 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.225 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.225 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.225 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.225 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.225 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.225 INFO:tasks.workunit.client.0.smithi096.stdout:raw2=.data-pool:marker___myobj_ 2023-09-22T13:23:28.226 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.226 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.226 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.226 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns__myobj_", 2023-09-22T13:23:28.226 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.226 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.226 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.226 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.226 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.226 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.226 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.226 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.227 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.227 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.227 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.227 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.227 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.227 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.227 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.227 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.227 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.227 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.227 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.227 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.228 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.228 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.228 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.228 INFO:tasks.workunit.client.0.smithi096.stdout:raw=.data-pool:marker__ns__myobj_ 2023-09-22T13:23:28.228 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.228 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj2": { 2023-09-22T13:23:28.228 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.228 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns__myobj_", 2023-09-22T13:23:28.228 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.228 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.228 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.228 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.229 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj2": { 2023-09-22T13:23:28.229 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.229 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.229 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.229 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.229 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.229 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.229 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.229 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.229 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.229 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.229 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.229 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.230 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.230 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "", 2023-09-22T13:23:28.230 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.230 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.230 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.230 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.230 INFO:tasks.workunit.client.0.smithi096.stdout:raw2=.data-pool:marker__ns__myobj_ 2023-09-22T13:23:28.230 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.230 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.230 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.230 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__:inst__myobj_", 2023-09-22T13:23:28.231 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "marker__myobj_" 2023-09-22T13:23:28.231 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.231 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.231 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.231 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.231 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.231 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.231 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.231 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.231 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.231 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.231 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.231 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.232 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.232 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.232 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.232 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.232 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.232 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.232 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.232 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.232 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.232 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.232 INFO:tasks.workunit.client.0.smithi096.stdout:raw=.data-pool:marker__:inst__myobj_ 2023-09-22T13:23:28.233 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.233 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj2": { 2023-09-22T13:23:28.233 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.233 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__:inst__myobj_", 2023-09-22T13:23:28.233 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "marker__myobj_" 2023-09-22T13:23:28.233 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.233 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.233 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.233 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj2": { 2023-09-22T13:23:28.233 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.233 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.233 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.233 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.234 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.234 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.234 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.234 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.234 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.234 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.234 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.234 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.234 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.234 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.234 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "" 2023-09-22T13:23:28.235 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.235 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.235 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.235 INFO:tasks.workunit.client.0.smithi096.stdout:raw2=.data-pool:marker__:inst__myobj_ 2023-09-22T13:23:28.235 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.235 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj": { 2023-09-22T13:23:28.235 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.235 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns:inst__myobj_", 2023-09-22T13:23:28.235 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.235 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.235 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.235 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.235 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj": { 2023-09-22T13:23:28.236 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.236 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.236 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.236 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.236 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.236 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.236 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.236 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.236 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.236 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.236 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.236 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.237 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.237 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.237 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.237 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.237 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.237 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.237 INFO:tasks.workunit.client.0.smithi096.stdout:raw=.data-pool:marker__ns:inst__myobj_ 2023-09-22T13:23:28.237 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.237 INFO:tasks.workunit.client.0.smithi096.stdout: "raw_obj2": { 2023-09-22T13:23:28.237 INFO:tasks.workunit.client.0.smithi096.stdout: "pool": ".data-pool", 2023-09-22T13:23:28.237 INFO:tasks.workunit.client.0.smithi096.stdout: "oid": "marker__ns:inst__myobj_", 2023-09-22T13:23:28.237 INFO:tasks.workunit.client.0.smithi096.stdout: "loc": "" 2023-09-22T13:23:28.237 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.238 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.238 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:28.238 INFO:tasks.workunit.client.0.smithi096.stdout: "new_obj2": { 2023-09-22T13:23:28.238 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket": { 2023-09-22T13:23:28.238 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "ebtest", 2023-09-22T13:23:28.238 INFO:tasks.workunit.client.0.smithi096.stdout: "marker": "marker", 2023-09-22T13:23:28.238 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_id": "bucket-id", 2023-09-22T13:23:28.238 INFO:tasks.workunit.client.0.smithi096.stdout: "tenant": "", 2023-09-22T13:23:28.238 INFO:tasks.workunit.client.0.smithi096.stdout: "explicit_placement": { 2023-09-22T13:23:28.238 INFO:tasks.workunit.client.0.smithi096.stdout: "data_pool": ".data-pool", 2023-09-22T13:23:28.238 INFO:tasks.workunit.client.0.smithi096.stdout: "data_extra_pool": "", 2023-09-22T13:23:28.238 INFO:tasks.workunit.client.0.smithi096.stdout: "index_pool": ".index-pool" 2023-09-22T13:23:28.239 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.239 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:28.239 INFO:tasks.workunit.client.0.smithi096.stdout: "key": { 2023-09-22T13:23:28.239 INFO:tasks.workunit.client.0.smithi096.stdout: "name": "_myobj_", 2023-09-22T13:23:28.239 INFO:tasks.workunit.client.0.smithi096.stdout: "instance": "inst", 2023-09-22T13:23:28.239 INFO:tasks.workunit.client.0.smithi096.stdout: "ns": "ns" 2023-09-22T13:23:28.239 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.239 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:28.239 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:28.239 INFO:tasks.workunit.client.0.smithi096.stdout:raw2=.data-pool:marker__ns:inst__myobj_ 2023-09-22T13:23:28.239 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] TestRGWObj.old_to_raw (0 ms) 2023-09-22T13:23:28.239 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 4 tests from TestRGWObj (1 ms total) 2023-09-22T13:23:28.240 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:23:28.240 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:23:28.240 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 4 tests from 1 test suite ran. (1 ms total) 2023-09-22T13:23:28.240 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 4 tests. 2023-09-22T13:23:28.240 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-09-22T13:23:28.240 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-22T13:23:28.272 INFO:tasks.workunit:Running workunits matching rgw/test_rgw_throttle.sh on client.0... 2023-09-22T13:23:28.273 INFO:tasks.workunit:Running workunit rgw/test_rgw_throttle.sh... 2023-09-22T13:23:28.274 DEBUG:teuthology.orchestra.run.smithi096:workunit test rgw/test_rgw_throttle.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=31c126b7c1c0c06b66d9d19c5bfc51c46b503392 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_throttle.sh 2023-09-22T13:23:28.356 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:23:28.357 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:23:28.357 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 5 tests from 1 test suite. 2023-09-22T13:23:28.357 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:23:29.913 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 5 tests from Aio_Throttle 2023-09-22T13:23:29.913 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] Aio_Throttle.NoThrottleUpToMax 2023-09-22T13:23:29.913 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] Aio_Throttle.NoThrottleUpToMax (0 ms) 2023-09-22T13:23:29.913 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] Aio_Throttle.CostOverWindow 2023-09-22T13:23:29.913 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] Aio_Throttle.CostOverWindow (0 ms) 2023-09-22T13:23:29.914 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] Aio_Throttle.ThrottleOverMax 2023-09-22T13:23:30.000 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] Aio_Throttle.ThrottleOverMax (88 ms) 2023-09-22T13:23:30.001 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] Aio_Throttle.YieldCostOverWindow 2023-09-22T13:23:30.001 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] Aio_Throttle.YieldCostOverWindow (0 ms) 2023-09-22T13:23:30.001 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] Aio_Throttle.YieldingThrottleOverMax 2023-09-22T13:23:30.089 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] Aio_Throttle.YieldingThrottleOverMax (88 ms) 2023-09-22T13:23:30.089 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 5 tests from Aio_Throttle (176 ms total) 2023-09-22T13:23:30.089 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:23:30.090 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:23:30.090 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 5 tests from 1 test suite ran. (1734 ms total) 2023-09-22T13:23:30.090 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 5 tests. 2023-09-22T13:23:30.093 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-09-22T13:23:30.093 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-22T13:23:30.124 INFO:tasks.workunit:Running workunits matching rgw/test_librgw_file.sh on client.0... 2023-09-22T13:23:30.125 INFO:tasks.workunit:Running workunit rgw/test_librgw_file.sh... 2023-09-22T13:23:30.125 DEBUG:teuthology.orchestra.run.smithi096:workunit test rgw/test_librgw_file.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=31c126b7c1c0c06b66d9d19c5bfc51c46b503392 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh 2023-09-22T13:23:30.329 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:23:30.329 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:23:30.337 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.336+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.337+0000 7f1ba54aa500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:30.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.337+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.339 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.338+0000 7f1ba54aa500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:30.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.338+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.338+0000 7f1ba54aa500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:30.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.338+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.340 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.339+0000 7f1ba54aa500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:23:30.353 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.351+0000 7f1ba54aa500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:23:30.353 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.351+0000 7f1ba54aa500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:23:30.353 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.351+0000 7f1ba54aa500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:23:30.354 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.351+0000 7f1ba54aa500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:23:30.354 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.351+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.354 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.352+0000 7f1ba54aa500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:23:30.354 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.352+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.354 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.352+0000 7f1ba54aa500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:23:30.354 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.352+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.356 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.352+0000 7f1ba54aa500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:23:30.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.352+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.353+0000 7f1ba54aa500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:23:30.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.353+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.353+0000 7f1ba54aa500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:30.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.353+0000 7f1ba54aa500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:23:30.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.353+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.353+0000 7f1ba54aa500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:30.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.353+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.353+0000 7f1ba54aa500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:23:30.357 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.353+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.354+0000 7f1ba54aa500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:23:30.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.354+0000 7f1ba54aa500 20 zonegroup default 2023-09-22T13:23:30.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.354+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.354+0000 7f1ba54aa500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:30.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.354+0000 7f1ba54aa500 10 Cannot find current period zone using local zone 2023-09-22T13:23:30.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.354+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.354+0000 7f1ba54aa500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:30.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.354+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.358 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.354+0000 7f1ba54aa500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:23:30.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.354+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.355+0000 7f1ba54aa500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:23:30.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.355+0000 7f1ba54aa500 20 zone default found 2023-09-22T13:23:30.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.355+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.355+0000 7f1ba54aa500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:30.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.355+0000 7f1ba54aa500 20 rados->read ofs=0 len=0 2023-09-22T13:23:30.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.355+0000 7f1ba54aa500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:30.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.355+0000 7f1ba54aa500 20 started sync module instance, tier type = 2023-09-22T13:23:30.359 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.355+0000 7f1ba54aa500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:23:30.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.363+0000 7f1ba54aa500 20 add_watcher() i=0 2023-09-22T13:23:30.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.363+0000 7f1ba54aa500 20 add_watcher() i=1 2023-09-22T13:23:30.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.363+0000 7f1ba54aa500 20 add_watcher() i=2 2023-09-22T13:23:30.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.363+0000 7f1ba54aa500 20 add_watcher() i=3 2023-09-22T13:23:30.364 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.363+0000 7f1ba54aa500 20 add_watcher() i=4 2023-09-22T13:23:30.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.363+0000 7f1ba54aa500 20 add_watcher() i=5 2023-09-22T13:23:30.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.363+0000 7f1ba54aa500 20 add_watcher() i=6 2023-09-22T13:23:30.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.363+0000 7f1b9098c700 0 -- 172.21.15.96:0/3338984676 >> [v2:172.21.15.96:6807/140797,v1:172.21.15.96:6809/140797] conn(0x7f1b840156b0 msgr2=0x7f1b84015ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:23:30.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.363+0000 7f1ba54aa500 20 add_watcher() i=7 2023-09-22T13:23:30.365 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.363+0000 7f1ba54aa500 2 all 8 watchers are set, enabling cache 2023-09-22T13:23:30.366 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.365+0000 7f1b61fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:23:30.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.365+0000 7f1ba54aa500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:23:30.367 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.365+0000 7f1ba54aa500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:23:30.370 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.369+0000 7f1b8bfff700 0 -- 172.21.15.96:0/2774747737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555a94cdf310 msgr2=0x555a94cf1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:23:30.571 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.571+0000 7f1b8bfff700 0 --2- 172.21.15.96:0/2774747737 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x555a94cdf310 0x555a94cf1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-09-22T13:23:30.573 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.573+0000 7f1ba54aa500 5 note: GC not initialized 2023-09-22T13:23:30.574 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.574+0000 7f1ad8ff9700 20 reqs_thread_entry: start 2023-09-22T13:23:30.575 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.574+0000 7f1ace7fc700 20 reqs_thread_entry: start 2023-09-22T13:23:30.575 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.574+0000 7f1ba54aa500 20 init_complete bucket index max shards: 11 2023-09-22T13:23:30.575 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.575+0000 7f1ba54aa500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:23:30.575 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.575+0000 7f1accff9700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:24:00 2023 2023-09-22T13:23:30.575 INFO:tasks.workunit.client.0.smithi096.stderr: 2023-09-22T13:23:30.576 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.575+0000 7f1ab3fff700 20 reqs_thread_entry: start 2023-09-22T13:23:30.576 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.575+0000 7f1ba54aa500 20 get_system_obj_state: rctx=0x7fff6c827c88 obj=default.rgw.meta:users.uid:ceph-test-librgw-file state=0x555a94b27810 s->prefetch_data=0 2023-09-22T13:23:30.576 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.575+0000 7f1ba54aa500 10 cache get: name=default.rgw.meta+users.uid+ceph-test-librgw-file : miss 2023-09-22T13:23:30.577 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.575+0000 7f1ba54aa500 10 cache put: name=default.rgw.meta+users.uid+ceph-test-librgw-file info.flags=0x0 2023-09-22T13:23:30.577 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.575+0000 7f1ba54aa500 10 adding default.rgw.meta+users.uid+ceph-test-librgw-file to cache LRU end 2023-09-22T13:23:30.577 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.575+0000 7f1ba54aa500 20 get_system_obj_state: rctx=0x7fff6c827c98 obj=default.rgw.meta:users.email:librgw@example.com state=0x555a94b27810 s->prefetch_data=0 2023-09-22T13:23:30.577 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.575+0000 7f1ba54aa500 10 cache get: name=default.rgw.meta+users.email+librgw@example.com : miss 2023-09-22T13:23:30.577 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.576+0000 7f1ba54aa500 10 cache put: name=default.rgw.meta+users.email+librgw@example.com info.flags=0x0 2023-09-22T13:23:30.577 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.576+0000 7f1ba54aa500 10 adding default.rgw.meta+users.email+librgw@example.com to cache LRU end 2023-09-22T13:23:30.577 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.576+0000 7f1ba54aa500 20 get_system_obj_state: rctx=0x7fff6c827c98 obj=default.rgw.meta:users.keys:K7Mrz6J38oC8dtG9R+qSpOoK6dw= state=0x555a94b27810 s->prefetch_data=0 2023-09-22T13:23:30.577 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.576+0000 7f1ba54aa500 10 cache get: name=default.rgw.meta+users.keys+K7Mrz6J38oC8dtG9R+qSpOoK6dw= : miss 2023-09-22T13:23:30.577 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.576+0000 7f1ba54aa500 10 cache put: name=default.rgw.meta+users.keys+K7Mrz6J38oC8dtG9R+qSpOoK6dw= info.flags=0x0 2023-09-22T13:23:30.577 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.576+0000 7f1ba54aa500 10 adding default.rgw.meta+users.keys+K7Mrz6J38oC8dtG9R+qSpOoK6dw= to cache LRU end 2023-09-22T13:23:30.577 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.576+0000 7f1ba54aa500 20 get_system_obj_state: rctx=0x7fff6c827598 obj=default.rgw.meta:users.keys:K7Mrz6J38oC8dtG9R+qSpOoK6dw= state=0x555a94b27810 s->prefetch_data=0 2023-09-22T13:23:30.578 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.576+0000 7f1ba54aa500 10 cache get: name=default.rgw.meta+users.keys+K7Mrz6J38oC8dtG9R+qSpOoK6dw= : hit (negative entry) 2023-09-22T13:23:30.578 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.576+0000 7f1ba54aa500 20 get_system_obj_state: rctx=0x7fff6c8278d8 obj=default.rgw.meta:users.keys:K7Mrz6J38oC8dtG9R+qSpOoK6dw= state=0x555a94b27810 s->prefetch_data=0 2023-09-22T13:23:30.578 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.576+0000 7f1ba54aa500 10 cache get: name=default.rgw.meta+users.keys+K7Mrz6J38oC8dtG9R+qSpOoK6dw= : hit (negative entry) 2023-09-22T13:23:30.578 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.577+0000 7f1ba54aa500 10 cache put: name=default.rgw.meta+users.uid+ceph-test-librgw-file info.flags=0x17 2023-09-22T13:23:30.578 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.577+0000 7f1ba54aa500 10 moving default.rgw.meta+users.uid+ceph-test-librgw-file to cache LRU end 2023-09-22T13:23:30.579 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.577+0000 7f1ba54aa500 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.uid:ceph-test-librgw-file, ofs0, ns] 2023-09-22T13:23:30.579 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.578+0000 7f1b8a7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 987842478081 cookie 93847531706240 notifier 5316 bl.length()=663 2023-09-22T13:23:30.579 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.578+0000 7f1b8a7fc700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ceph-test-librgw-file info.flags=0x17 2023-09-22T13:23:30.579 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.578+0000 7f1b8a7fc700 10 rgw watcher librados: moving default.rgw.meta+users.uid+ceph-test-librgw-file to cache LRU end 2023-09-22T13:23:30.579 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.579+0000 7f1ba54aa500 10 cache put: name=default.rgw.meta+users.email+librgw@example.com info.flags=0x7 2023-09-22T13:23:30.580 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.579+0000 7f1ba54aa500 10 moving default.rgw.meta+users.email+librgw@example.com to cache LRU end 2023-09-22T13:23:30.580 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.579+0000 7f1ba54aa500 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:librgw@example.com, ofs0, ns] 2023-09-22T13:23:30.580 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.579+0000 7f1b89ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 987842478080 cookie 93847531709952 notifier 5316 bl.length()=194 2023-09-22T13:23:30.580 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.579+0000 7f1b89ffb700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+librgw@example.com info.flags=0x7 2023-09-22T13:23:30.580 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.579+0000 7f1b89ffb700 10 rgw watcher librados: moving default.rgw.meta+users.email+librgw@example.com to cache LRU end 2023-09-22T13:23:30.581 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.581+0000 7f1ba54aa500 10 cache put: name=default.rgw.meta+users.keys+K7Mrz6J38oC8dtG9R+qSpOoK6dw= info.flags=0x7 2023-09-22T13:23:30.582 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.581+0000 7f1ba54aa500 10 moving default.rgw.meta+users.keys+K7Mrz6J38oC8dtG9R+qSpOoK6dw= to cache LRU end 2023-09-22T13:23:30.582 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.581+0000 7f1ba54aa500 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:K7Mrz6J38oC8dtG9R+qSpOoK6dw=, ofs0, ns] 2023-09-22T13:23:30.582 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.581+0000 7f1b89ffb700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 987842478080 cookie 93847531721200 notifier 5316 bl.length()=203 2023-09-22T13:23:30.582 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.581+0000 7f1b89ffb700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+K7Mrz6J38oC8dtG9R+qSpOoK6dw= info.flags=0x7 2023-09-22T13:23:30.582 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.581+0000 7f1b89ffb700 10 rgw watcher librados: moving default.rgw.meta+users.keys+K7Mrz6J38oC8dtG9R+qSpOoK6dw= to cache LRU end 2023-09-22T13:23:30.582 INFO:tasks.workunit.client.0.smithi096.stdout:{ 2023-09-22T13:23:30.582 INFO:tasks.workunit.client.0.smithi096.stdout: "user_id": "ceph-test-librgw-file", 2023-09-22T13:23:30.583 INFO:tasks.workunit.client.0.smithi096.stdout: "display_name": "librgw test user", 2023-09-22T13:23:30.583 INFO:tasks.workunit.client.0.smithi096.stdout: "email": "librgw@example.com", 2023-09-22T13:23:30.583 INFO:tasks.workunit.client.0.smithi096.stdout: "suspended": 0, 2023-09-22T13:23:30.583 INFO:tasks.workunit.client.0.smithi096.stdout: "max_buckets": 1000, 2023-09-22T13:23:30.583 INFO:tasks.workunit.client.0.smithi096.stdout: "subusers": [], 2023-09-22T13:23:30.583 INFO:tasks.workunit.client.0.smithi096.stdout: "keys": [ 2023-09-22T13:23:30.583 INFO:tasks.workunit.client.0.smithi096.stdout: { 2023-09-22T13:23:30.583 INFO:tasks.workunit.client.0.smithi096.stdout: "user": "ceph-test-librgw-file", 2023-09-22T13:23:30.583 INFO:tasks.workunit.client.0.smithi096.stdout: "access_key": "K7Mrz6J38oC8dtG9R+qSpOoK6dw=", 2023-09-22T13:23:30.583 INFO:tasks.workunit.client.0.smithi096.stdout: "secret_key": "rZAl44knZGB7kHXv+Wdvy6uMymkCjwjY0gRQi50skuZxoJLLRpn4Fg==" 2023-09-22T13:23:30.583 INFO:tasks.workunit.client.0.smithi096.stdout: } 2023-09-22T13:23:30.584 INFO:tasks.workunit.client.0.smithi096.stdout: ], 2023-09-22T13:23:30.584 INFO:tasks.workunit.client.0.smithi096.stdout: "swift_keys": [], 2023-09-22T13:23:30.584 INFO:tasks.workunit.client.0.smithi096.stdout: "caps": [], 2023-09-22T13:23:30.584 INFO:tasks.workunit.client.0.smithi096.stdout: "op_mask": "read, write, delete", 2023-09-22T13:23:30.584 INFO:tasks.workunit.client.0.smithi096.stdout: "default_placement": "", 2023-09-22T13:23:30.584 INFO:tasks.workunit.client.0.smithi096.stdout: "default_storage_class": "", 2023-09-22T13:23:30.584 INFO:tasks.workunit.client.0.smithi096.stdout: "placement_tags": [], 2023-09-22T13:23:30.584 INFO:tasks.workunit.client.0.smithi096.stdout: "bucket_quota": { 2023-09-22T13:23:30.584 INFO:tasks.workunit.client.0.smithi096.stdout: "enabled": false, 2023-09-22T13:23:30.584 INFO:tasks.workunit.client.0.smithi096.stdout: "check_on_raw": false, 2023-09-22T13:23:30.584 INFO:tasks.workunit.client.0.smithi096.stdout: "max_size": -1, 2023-09-22T13:23:30.585 INFO:tasks.workunit.client.0.smithi096.stdout: "max_size_kb": 0, 2023-09-22T13:23:30.585 INFO:tasks.workunit.client.0.smithi096.stdout: "max_objects": -1 2023-09-22T13:23:30.585 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:30.585 INFO:tasks.workunit.client.0.smithi096.stdout: "user_quota": { 2023-09-22T13:23:30.585 INFO:tasks.workunit.client.0.smithi096.stdout: "enabled": false, 2023-09-22T13:23:30.585 INFO:tasks.workunit.client.0.smithi096.stdout: "check_on_raw": false, 2023-09-22T13:23:30.585 INFO:tasks.workunit.client.0.smithi096.stdout: "max_size": -1, 2023-09-22T13:23:30.585 INFO:tasks.workunit.client.0.smithi096.stdout: "max_size_kb": 0, 2023-09-22T13:23:30.585 INFO:tasks.workunit.client.0.smithi096.stdout: "max_objects": -1 2023-09-22T13:23:30.585 INFO:tasks.workunit.client.0.smithi096.stdout: }, 2023-09-22T13:23:30.586 INFO:tasks.workunit.client.0.smithi096.stdout: "temp_url_keys": [], 2023-09-22T13:23:30.586 INFO:tasks.workunit.client.0.smithi096.stdout: "type": "rgw", 2023-09-22T13:23:30.586 INFO:tasks.workunit.client.0.smithi096.stdout: "mfa_ids": [] 2023-09-22T13:23:30.586 INFO:tasks.workunit.client.0.smithi096.stdout:} 2023-09-22T13:23:30.586 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:23:30.587 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.583+0000 7f1ba54aa500 20 remove_watcher() i=0 2023-09-22T13:23:30.587 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.583+0000 7f1ba54aa500 2 removed watcher, disabling cache 2023-09-22T13:23:30.587 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.583+0000 7f1ba54aa500 20 remove_watcher() i=1 2023-09-22T13:23:30.588 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.584+0000 7f1ba54aa500 20 remove_watcher() i=2 2023-09-22T13:23:30.588 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.585+0000 7f1ba54aa500 20 remove_watcher() i=3 2023-09-22T13:23:30.588 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.586+0000 7f1ba54aa500 20 remove_watcher() i=4 2023-09-22T13:23:30.588 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.587+0000 7f1ba54aa500 20 remove_watcher() i=5 2023-09-22T13:23:30.588 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.587+0000 7f1ba54aa500 20 remove_watcher() i=6 2023-09-22T13:23:30.588 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.588+0000 7f1ba54aa500 20 remove_watcher() i=7 2023-09-22T13:23:30.589 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:30.589+0000 7f1b61fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:23:30.594 INFO:tasks.workunit.client.0.smithi096.stdout:phase 1.1 2023-09-22T13:23:30.609 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 24 tests from 1 test suite. 2023-09-22T13:23:30.609 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:23:30.609 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 24 tests from LibRGW 2023-09-22T13:23:30.610 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.INIT 2023-09-22T13:23:30.620 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:23:30.620 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:23:30.961 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.INIT (351 ms) 2023-09-22T13:23:30.961 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MOUNT_NOROOT 2023-09-22T13:23:30.963 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MOUNT_NOROOT (2 ms) 2023-09-22T13:23:30.963 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MOUNT 2023-09-22T13:23:30.963 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MOUNT (0 ms) 2023-09-22T13:23:30.963 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SETUP_HIER1 2023-09-22T13:23:30.971 INFO:tasks.workunit.client.0.smithi096.stdout:creating: nfsroot:foo/bar/baz/quux 2023-09-22T13:23:30.975 INFO:tasks.workunit.client.0.smithi096.stdout:creating: nfsroot:foo/f1 2023-09-22T13:23:30.978 INFO:tasks.workunit.client.0.smithi096.stdout:creating: nfsroot:foo/f2 2023-09-22T13:23:30.980 INFO:tasks.workunit.client.0.smithi096.stdout:creating: nfsroot:foo/bar/f1 2023-09-22T13:23:30.983 INFO:tasks.workunit.client.0.smithi096.stdout:creating: nfsroot:foo/bar/d1/ 2023-09-22T13:23:30.985 INFO:tasks.workunit.client.0.smithi096.stdout:creating: nfsroot:foo/bar/baz/hungry 2023-09-22T13:23:30.987 INFO:tasks.workunit.client.0.smithi096.stdout:creating: nfsroot:foo/bar/baz/hungry/ 2023-09-22T13:23:30.990 INFO:tasks.workunit.client.0.smithi096.stdout:creating: nfsroot:foo/bar/baz/momma 2023-09-22T13:23:30.992 INFO:tasks.workunit.client.0.smithi096.stdout:creating: nfsroot:foo/bar/baz/bear/ 2023-09-22T13:23:30.995 INFO:tasks.workunit.client.0.smithi096.stdout:creating: nfsroot:foo/bar/baz/sasquatch 2023-09-22T13:23:30.997 INFO:tasks.workunit.client.0.smithi096.stdout:creating: nfsroot:foo/bar/baz/sasquatch/ 2023-09-22T13:23:31.000 INFO:tasks.workunit.client.0.smithi096.stdout:creating: nfsroot:foo/bar/baz/frobozz 2023-09-22T13:23:31.002 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SETUP_HIER1 (40 ms) 2023-09-22T13:23:31.002 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SETUP_DIRS1 2023-09-22T13:23:31.109 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SETUP_DIRS1 (106 ms) 2023-09-22T13:23:31.109 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SETATTR 2023-09-22T13:23:31.109 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SETATTR (0 ms) 2023-09-22T13:23:31.109 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.RGW_CREATE_DIRS1 2023-09-22T13:23:31.129 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.RGW_CREATE_DIRS1 (20 ms) 2023-09-22T13:23:31.129 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.RGW_SETUP_RENAME1 2023-09-22T13:23:31.166 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.RGW_SETUP_RENAME1 (38 ms) 2023-09-22T13:23:31.167 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.RGW_INTRABUCKET_RENAME1 2023-09-22T13:23:31.167 INFO:tasks.workunit.client.0.smithi096.stdout:rename file rfile_0 to rfile_r0 (bucket brename0) 2023-09-22T13:23:31.171 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.RGW_INTRABUCKET_RENAME1 (5 ms) 2023-09-22T13:23:31.171 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.RGW_CROSSBUCKET_RENAME1 2023-09-22T13:23:31.171 INFO:tasks.workunit.client.0.smithi096.stdout:rename file rfile_1 (bucket brename0) to rfile_rhilldog (bucket brename1) 2023-09-22T13:23:31.176 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.RGW_CROSSBUCKET_RENAME1 (5 ms) 2023-09-22T13:23:31.176 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.GETATTR_DIRS1 2023-09-22T13:23:31.177 INFO:tasks.workunit.client.0.smithi096.stdout:scanning objects in /bdirs1/dir_0 2023-09-22T13:23:31.177 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_0/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-09-22T13:23:31.177 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_0/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-09-22T13:23:31.177 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_0/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-09-22T13:23:31.177 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_0/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-09-22T13:23:31.177 INFO:tasks.workunit.client.0.smithi096.stdout:scanning objects in /bdirs1/dir_1 2023-09-22T13:23:31.177 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_1/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-09-22T13:23:31.177 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_1/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-09-22T13:23:31.177 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_1/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-09-22T13:23:31.177 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_1/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-09-22T13:23:31.177 INFO:tasks.workunit.client.0.smithi096.stdout:scanning objects in /bdirs1/dir_2 2023-09-22T13:23:31.178 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_2/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-09-22T13:23:31.178 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_2/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-09-22T13:23:31.178 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_2/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-09-22T13:23:31.178 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_2/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-09-22T13:23:31.178 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.GETATTR_DIRS1 (0 ms) 2023-09-22T13:23:31.178 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.READ_DIRS1 2023-09-22T13:23:31.178 INFO:tasks.workunit.client.0.smithi096.stdout:read back objects in /bdirs1/dir_0 2023-09-22T13:23:31.178 INFO:tasks.workunit.client.0.smithi096.stdout:reading 0,256 dir_0/sfile_0 2023-09-22T13:23:31.179 INFO:tasks.workunit.client.0.smithi096.stdout: read back from sfile_0 : "data for sfile_0" 2023-09-22T13:23:31.179 INFO:tasks.workunit.client.0.smithi096.stdout:reading 0,256 dir_0/sfile_1 2023-09-22T13:23:31.179 INFO:tasks.workunit.client.0.smithi096.stdout: read back from sfile_1 : "data for sfile_1" 2023-09-22T13:23:31.179 INFO:tasks.workunit.client.0.smithi096.stdout:read back objects in /bdirs1/dir_1 2023-09-22T13:23:31.179 INFO:tasks.workunit.client.0.smithi096.stdout:reading 0,256 dir_1/sfile_0 2023-09-22T13:23:31.179 INFO:tasks.workunit.client.0.smithi096.stdout: read back from sfile_0 : "data for sfile_0" 2023-09-22T13:23:31.179 INFO:tasks.workunit.client.0.smithi096.stdout:reading 0,256 dir_1/sfile_1 2023-09-22T13:23:31.180 INFO:tasks.workunit.client.0.smithi096.stdout: read back from sfile_1 : "data for sfile_1" 2023-09-22T13:23:31.181 INFO:tasks.workunit.client.0.smithi096.stdout:read back objects in /bdirs1/dir_2 2023-09-22T13:23:31.181 INFO:tasks.workunit.client.0.smithi096.stdout:reading 0,256 dir_2/sfile_0 2023-09-22T13:23:31.182 INFO:tasks.workunit.client.0.smithi096.stdout: read back from sfile_0 : "data for sfile_0" 2023-09-22T13:23:31.182 INFO:tasks.workunit.client.0.smithi096.stdout:reading 0,256 dir_2/sfile_1 2023-09-22T13:23:31.183 INFO:tasks.workunit.client.0.smithi096.stdout: read back from sfile_1 : "data for sfile_1" 2023-09-22T13:23:31.183 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.READ_DIRS1 (6 ms) 2023-09-22T13:23:31.183 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.READF_SETUP1 2023-09-22T13:23:31.360 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.READF_SETUP1 (177 ms) 2023-09-22T13:23:31.360 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.READF_DIRS1 2023-09-22T13:23:31.360 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.READF_DIRS1 (0 ms) 2023-09-22T13:23:31.361 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.WRITEF_DIRS1 2023-09-22T13:23:31.361 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.WRITEF_DIRS1 (0 ms) 2023-09-22T13:23:31.361 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.RELEASE_DIRS1 2023-09-22T13:23:31.361 INFO:tasks.workunit.client.0.smithi096.stdout:release sdir_0 type: DIR refs: 2 2023-09-22T13:23:31.361 INFO:tasks.workunit.client.0.smithi096.stdout:release sfile_0 type: FILE refs: 2 2023-09-22T13:23:31.361 INFO:tasks.workunit.client.0.smithi096.stdout:release sdir_1 type: DIR refs: 2 2023-09-22T13:23:31.361 INFO:tasks.workunit.client.0.smithi096.stdout:release sfile_1 type: FILE refs: 2 2023-09-22T13:23:31.361 INFO:tasks.workunit.client.0.smithi096.stdout:release sdir_0 type: DIR refs: 2 2023-09-22T13:23:31.361 INFO:tasks.workunit.client.0.smithi096.stdout:release sfile_0 type: FILE refs: 2 2023-09-22T13:23:31.361 INFO:tasks.workunit.client.0.smithi096.stdout:release sdir_1 type: DIR refs: 2 2023-09-22T13:23:31.361 INFO:tasks.workunit.client.0.smithi096.stdout:release sfile_1 type: FILE refs: 2 2023-09-22T13:23:31.361 INFO:tasks.workunit.client.0.smithi096.stdout:release sdir_0 type: DIR refs: 2 2023-09-22T13:23:31.362 INFO:tasks.workunit.client.0.smithi096.stdout:release sfile_0 type: FILE refs: 2 2023-09-22T13:23:31.362 INFO:tasks.workunit.client.0.smithi096.stdout:release sdir_1 type: DIR refs: 2 2023-09-22T13:23:31.362 INFO:tasks.workunit.client.0.smithi096.stdout:release sfile_1 type: FILE refs: 2 2023-09-22T13:23:31.362 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.RELEASE_DIRS1 (1 ms) 2023-09-22T13:23:31.362 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.HIER1 2023-09-22T13:23:31.368 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/f2 (f2): FILE 2023-09-22T13:23:31.368 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/f1 (f1): FILE 2023-09-22T13:23:31.372 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/f1 (f1): FILE 2023-09-22T13:23:31.376 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/d1 (d1): DIR 2023-09-22T13:23:31.378 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz/sasquatch (sasquatch): FILE 2023-09-22T13:23:31.379 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz/sasquatch (sasquatch): FILE 2023-09-22T13:23:31.379 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz/quux (quux): FILE 2023-09-22T13:23:31.380 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz/momma (momma): FILE 2023-09-22T13:23:31.381 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz/hungry (hungry): FILE 2023-09-22T13:23:31.381 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz/hungry (hungry): FILE 2023-09-22T13:23:31.382 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz/frobozz (frobozz): FILE 2023-09-22T13:23:31.385 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz/bear (bear): DIR 2023-09-22T13:23:31.385 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz (baz): DIR 2023-09-22T13:23:31.385 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar (bar): DIR 2023-09-22T13:23:31.385 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo (foo): DIR 2023-09-22T13:23:31.385 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot (nfsroot): DIR 2023-09-22T13:23:31.385 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.HIER1 (24 ms) 2023-09-22T13:23:31.385 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2023-09-22T13:23:31.385 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (0 ms) 2023-09-22T13:23:31.386 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2023-09-22T13:23:31.386 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (0 ms) 2023-09-22T13:23:31.386 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2023-09-22T13:23:31.386 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_READDIR (0 ms) 2023-09-22T13:23:31.386 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2023-09-22T13:23:31.386 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2023-09-22T13:23:31.386 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CLEANUP 2023-09-22T13:23:31.386 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-09-22T13:23:31.386 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.UMOUNT 2023-09-22T13:23:31.386 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-09-22T13:23:31.386 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-09-22T13:23:31.387 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:31.384+0000 7f083cee0540 -1 shutting down 2023-09-22T13:23:31.387 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:31.384+0000 7f0750ff9700 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 148407, si_uid : 1000, si_addr0x3e8000243b7, si_status0 2023-09-22T13:23:31.462 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SHUTDOWN (78 ms) 2023-09-22T13:23:31.462 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 24 tests from LibRGW (853 ms total) 2023-09-22T13:23:31.463 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:23:31.463 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:23:31.463 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 24 tests from 1 test suite ran. (853 ms total) 2023-09-22T13:23:31.463 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 24 tests. 2023-09-22T13:23:31.464 INFO:tasks.workunit.client.0.smithi096.stdout:phase 1.2 2023-09-22T13:23:31.479 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 24 tests from 1 test suite. 2023-09-22T13:23:31.480 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:23:31.480 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 24 tests from LibRGW 2023-09-22T13:23:31.480 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.INIT 2023-09-22T13:23:31.487 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:23:31.487 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:23:31.572 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.INIT (92 ms) 2023-09-22T13:23:31.572 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MOUNT_NOROOT 2023-09-22T13:23:31.574 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MOUNT_NOROOT (2 ms) 2023-09-22T13:23:31.574 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MOUNT 2023-09-22T13:23:31.575 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MOUNT (0 ms) 2023-09-22T13:23:31.575 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SETUP_HIER1 2023-09-22T13:23:31.575 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SETUP_HIER1 (0 ms) 2023-09-22T13:23:31.575 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SETUP_DIRS1 2023-09-22T13:23:31.575 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SETUP_DIRS1 (0 ms) 2023-09-22T13:23:31.575 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SETATTR 2023-09-22T13:23:31.575 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SETATTR (0 ms) 2023-09-22T13:23:31.575 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.RGW_CREATE_DIRS1 2023-09-22T13:23:31.575 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.RGW_CREATE_DIRS1 (0 ms) 2023-09-22T13:23:31.575 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.RGW_SETUP_RENAME1 2023-09-22T13:23:31.576 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.RGW_SETUP_RENAME1 (0 ms) 2023-09-22T13:23:31.576 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.RGW_INTRABUCKET_RENAME1 2023-09-22T13:23:31.576 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.RGW_INTRABUCKET_RENAME1 (0 ms) 2023-09-22T13:23:31.576 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.RGW_CROSSBUCKET_RENAME1 2023-09-22T13:23:31.576 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.RGW_CROSSBUCKET_RENAME1 (0 ms) 2023-09-22T13:23:31.576 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.GETATTR_DIRS1 2023-09-22T13:23:31.577 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.GETATTR_DIRS1 (0 ms) 2023-09-22T13:23:31.577 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.READ_DIRS1 2023-09-22T13:23:31.577 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.READ_DIRS1 (0 ms) 2023-09-22T13:23:31.577 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.READF_SETUP1 2023-09-22T13:23:31.577 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.READF_SETUP1 (0 ms) 2023-09-22T13:23:31.577 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.READF_DIRS1 2023-09-22T13:23:31.577 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.READF_DIRS1 (0 ms) 2023-09-22T13:23:31.577 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.WRITEF_DIRS1 2023-09-22T13:23:31.577 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.WRITEF_DIRS1 (0 ms) 2023-09-22T13:23:31.578 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.RELEASE_DIRS1 2023-09-22T13:23:31.578 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.RELEASE_DIRS1 (0 ms) 2023-09-22T13:23:31.578 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.HIER1 2023-09-22T13:23:31.578 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.HIER1 (0 ms) 2023-09-22T13:23:31.578 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2023-09-22T13:23:31.578 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (0 ms) 2023-09-22T13:23:31.578 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2023-09-22T13:23:31.578 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (0 ms) 2023-09-22T13:23:31.578 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2023-09-22T13:23:31.578 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_READDIR (0 ms) 2023-09-22T13:23:31.579 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2023-09-22T13:23:31.579 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2023-09-22T13:23:31.579 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CLEANUP 2023-09-22T13:23:31.579 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-09-22T13:23:31.579 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.UMOUNT 2023-09-22T13:23:31.579 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-09-22T13:23:31.579 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-09-22T13:23:31.579 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:31.573+0000 7f5c1763c540 -1 shutting down 2023-09-22T13:23:31.580 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:31.573+0000 7f5b1effd700 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 148491, si_uid : 1000, si_addr0x3e80002440b, si_status0 2023-09-22T13:23:31.672 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SHUTDOWN (99 ms) 2023-09-22T13:23:31.673 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 24 tests from LibRGW (193 ms total) 2023-09-22T13:23:31.673 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:23:31.673 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:23:31.673 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 24 tests from 1 test suite ran. (193 ms total) 2023-09-22T13:23:31.673 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 24 tests. 2023-09-22T13:23:31.675 INFO:tasks.workunit.client.0.smithi096.stdout:phase 1.3 2023-09-22T13:23:31.690 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 24 tests from 1 test suite. 2023-09-22T13:23:31.691 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:23:31.691 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 24 tests from LibRGW 2023-09-22T13:23:31.691 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.INIT 2023-09-22T13:23:31.700 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:23:31.700 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:23:31.788 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.INIT (97 ms) 2023-09-22T13:23:31.788 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MOUNT_NOROOT 2023-09-22T13:23:31.789 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MOUNT_NOROOT (2 ms) 2023-09-22T13:23:31.789 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MOUNT 2023-09-22T13:23:31.789 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MOUNT (0 ms) 2023-09-22T13:23:31.789 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SETUP_HIER1 2023-09-22T13:23:31.794 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SETUP_HIER1 (5 ms) 2023-09-22T13:23:31.794 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SETUP_DIRS1 2023-09-22T13:23:31.853 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SETUP_DIRS1 (58 ms) 2023-09-22T13:23:31.853 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SETATTR 2023-09-22T13:23:31.853 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SETATTR (0 ms) 2023-09-22T13:23:31.853 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.RGW_CREATE_DIRS1 2023-09-22T13:23:31.853 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.RGW_CREATE_DIRS1 (0 ms) 2023-09-22T13:23:31.854 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.RGW_SETUP_RENAME1 2023-09-22T13:23:31.854 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.RGW_SETUP_RENAME1 (0 ms) 2023-09-22T13:23:31.854 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.RGW_INTRABUCKET_RENAME1 2023-09-22T13:23:31.854 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.RGW_INTRABUCKET_RENAME1 (0 ms) 2023-09-22T13:23:31.854 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.RGW_CROSSBUCKET_RENAME1 2023-09-22T13:23:31.854 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.RGW_CROSSBUCKET_RENAME1 (0 ms) 2023-09-22T13:23:31.854 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.GETATTR_DIRS1 2023-09-22T13:23:31.854 INFO:tasks.workunit.client.0.smithi096.stdout:scanning objects in /bdirs1/dir_0 2023-09-22T13:23:31.854 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_0/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-09-22T13:23:31.854 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_0/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-09-22T13:23:31.855 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_0/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-09-22T13:23:31.855 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_0/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-09-22T13:23:31.855 INFO:tasks.workunit.client.0.smithi096.stdout:scanning objects in /bdirs1/dir_1 2023-09-22T13:23:31.855 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_1/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-09-22T13:23:31.855 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_1/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-09-22T13:23:31.855 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_1/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-09-22T13:23:31.855 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_1/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-09-22T13:23:31.855 INFO:tasks.workunit.client.0.smithi096.stdout:scanning objects in /bdirs1/dir_2 2023-09-22T13:23:31.855 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_2/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-09-22T13:23:31.856 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_2/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-09-22T13:23:31.856 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_2/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-09-22T13:23:31.856 INFO:tasks.workunit.client.0.smithi096.stdout: /bdirs1/dir_2/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-09-22T13:23:31.856 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.GETATTR_DIRS1 (0 ms) 2023-09-22T13:23:31.856 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.READ_DIRS1 2023-09-22T13:23:31.856 INFO:tasks.workunit.client.0.smithi096.stdout:read back objects in /bdirs1/dir_0 2023-09-22T13:23:31.856 INFO:tasks.workunit.client.0.smithi096.stdout:reading 0,256 dir_0/sfile_0 2023-09-22T13:23:31.857 INFO:tasks.workunit.client.0.smithi096.stdout: read back from sfile_0 : "data for sfile_0" 2023-09-22T13:23:31.857 INFO:tasks.workunit.client.0.smithi096.stdout:reading 0,256 dir_0/sfile_1 2023-09-22T13:23:31.857 INFO:tasks.workunit.client.0.smithi096.stdout: read back from sfile_1 : "data for sfile_1" 2023-09-22T13:23:31.857 INFO:tasks.workunit.client.0.smithi096.stdout:read back objects in /bdirs1/dir_1 2023-09-22T13:23:31.857 INFO:tasks.workunit.client.0.smithi096.stdout:reading 0,256 dir_1/sfile_0 2023-09-22T13:23:31.857 INFO:tasks.workunit.client.0.smithi096.stdout: read back from sfile_0 : "data for sfile_0" 2023-09-22T13:23:31.858 INFO:tasks.workunit.client.0.smithi096.stdout:reading 0,256 dir_1/sfile_1 2023-09-22T13:23:31.858 INFO:tasks.workunit.client.0.smithi096.stdout: read back from sfile_1 : "data for sfile_1" 2023-09-22T13:23:31.858 INFO:tasks.workunit.client.0.smithi096.stdout:read back objects in /bdirs1/dir_2 2023-09-22T13:23:31.858 INFO:tasks.workunit.client.0.smithi096.stdout:reading 0,256 dir_2/sfile_0 2023-09-22T13:23:31.861 INFO:tasks.workunit.client.0.smithi096.stdout: read back from sfile_0 : "data for sfile_0" 2023-09-22T13:23:31.861 INFO:tasks.workunit.client.0.smithi096.stdout:reading 0,256 dir_2/sfile_1 2023-09-22T13:23:31.861 INFO:tasks.workunit.client.0.smithi096.stdout: read back from sfile_1 : "data for sfile_1" 2023-09-22T13:23:31.861 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.READ_DIRS1 (8 ms) 2023-09-22T13:23:31.861 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.READF_SETUP1 2023-09-22T13:23:31.861 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.READF_SETUP1 (0 ms) 2023-09-22T13:23:31.861 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.READF_DIRS1 2023-09-22T13:23:31.862 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.READF_DIRS1 (0 ms) 2023-09-22T13:23:31.862 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.WRITEF_DIRS1 2023-09-22T13:23:31.862 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.WRITEF_DIRS1 (0 ms) 2023-09-22T13:23:31.862 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.RELEASE_DIRS1 2023-09-22T13:23:31.862 INFO:tasks.workunit.client.0.smithi096.stdout:release sdir_0 type: DIR refs: 2 2023-09-22T13:23:31.862 INFO:tasks.workunit.client.0.smithi096.stdout:release sfile_0 type: FILE refs: 2 2023-09-22T13:23:31.862 INFO:tasks.workunit.client.0.smithi096.stdout:release sdir_1 type: DIR refs: 2 2023-09-22T13:23:31.862 INFO:tasks.workunit.client.0.smithi096.stdout:release sfile_1 type: FILE refs: 2 2023-09-22T13:23:31.862 INFO:tasks.workunit.client.0.smithi096.stdout:release sdir_0 type: DIR refs: 2 2023-09-22T13:23:31.862 INFO:tasks.workunit.client.0.smithi096.stdout:release sfile_0 type: FILE refs: 2 2023-09-22T13:23:31.862 INFO:tasks.workunit.client.0.smithi096.stdout:release sdir_1 type: DIR refs: 2 2023-09-22T13:23:31.863 INFO:tasks.workunit.client.0.smithi096.stdout:release sfile_1 type: FILE refs: 2 2023-09-22T13:23:31.863 INFO:tasks.workunit.client.0.smithi096.stdout:release sdir_0 type: DIR refs: 2 2023-09-22T13:23:31.863 INFO:tasks.workunit.client.0.smithi096.stdout:release sfile_0 type: FILE refs: 2 2023-09-22T13:23:31.863 INFO:tasks.workunit.client.0.smithi096.stdout:release sdir_1 type: DIR refs: 2 2023-09-22T13:23:31.863 INFO:tasks.workunit.client.0.smithi096.stdout:release sfile_1 type: FILE refs: 2 2023-09-22T13:23:31.863 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.RELEASE_DIRS1 (0 ms) 2023-09-22T13:23:31.863 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.HIER1 2023-09-22T13:23:31.868 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/f2 (f2): FILE 2023-09-22T13:23:31.869 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/f1 (f1): FILE 2023-09-22T13:23:31.873 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/f1 (f1): FILE 2023-09-22T13:23:31.875 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/d1 (d1): DIR 2023-09-22T13:23:31.879 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz/sasquatch (sasquatch): FILE 2023-09-22T13:23:31.879 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz/sasquatch (sasquatch): FILE 2023-09-22T13:23:31.880 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz/quux (quux): FILE 2023-09-22T13:23:31.881 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz/momma (momma): FILE 2023-09-22T13:23:31.882 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz/hungry (hungry): FILE 2023-09-22T13:23:31.882 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz/hungry (hungry): FILE 2023-09-22T13:23:31.883 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz/frobozz (frobozz): FILE 2023-09-22T13:23:31.886 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz/bear (bear): DIR 2023-09-22T13:23:31.886 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar/baz (baz): DIR 2023-09-22T13:23:31.886 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo/bar (bar): DIR 2023-09-22T13:23:31.887 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot/foo (foo): DIR 2023-09-22T13:23:31.887 INFO:tasks.workunit.client.0.smithi096.stdout:/nfsroot (nfsroot): DIR 2023-09-22T13:23:31.887 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.HIER1 (25 ms) 2023-09-22T13:23:31.887 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2023-09-22T13:23:31.887 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (0 ms) 2023-09-22T13:23:31.888 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2023-09-22T13:23:31.888 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (0 ms) 2023-09-22T13:23:31.888 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2023-09-22T13:23:31.888 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_READDIR (0 ms) 2023-09-22T13:23:31.888 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2023-09-22T13:23:31.888 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2023-09-22T13:23:31.888 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CLEANUP 2023-09-22T13:23:31.888 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-09-22T13:23:31.888 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.UMOUNT 2023-09-22T13:23:31.889 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-09-22T13:23:31.889 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-09-22T13:23:31.889 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:31.885+0000 7fa261191540 -1 shutting down 2023-09-22T13:23:31.889 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:31.885+0000 7fa1757fa700 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 148575, si_uid : 1000, si_addr0x3e80002445f, si_status0 2023-09-22T13:23:31.989 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SHUTDOWN (104 ms) 2023-09-22T13:23:31.989 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 24 tests from LibRGW (299 ms total) 2023-09-22T13:23:31.989 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:23:31.990 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:23:31.990 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 24 tests from 1 test suite ran. (299 ms total) 2023-09-22T13:23:31.990 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 24 tests. 2023-09-22T13:23:31.991 INFO:tasks.workunit.client.0.smithi096.stdout:phase 2.1 2023-09-22T13:23:32.006 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 9 tests from 1 test suite. 2023-09-22T13:23:32.006 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:23:32.007 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 9 tests from LibRGW 2023-09-22T13:23:32.007 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.INIT 2023-09-22T13:23:32.015 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:23:32.015 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:23:32.089 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.INIT (82 ms) 2023-09-22T13:23:32.089 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MOUNT 2023-09-22T13:23:32.090 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MOUNT (1 ms) 2023-09-22T13:23:32.090 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-09-22T13:23:32.097 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CREATE_BUCKET (8 ms) 2023-09-22T13:23:32.097 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-09-22T13:23:32.097 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.DELETE_BUCKET (0 ms) 2023-09-22T13:23:32.098 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CREATE_BUCKET_MULTI 2023-09-22T13:23:32.105 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave0 2023-09-22T13:23:32.112 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave1 2023-09-22T13:23:32.119 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave2 2023-09-22T13:23:32.126 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave3 2023-09-22T13:23:32.134 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave4 2023-09-22T13:23:32.141 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave5 2023-09-22T13:23:32.148 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave6 2023-09-22T13:23:32.156 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave7 2023-09-22T13:23:32.162 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave8 2023-09-22T13:23:32.168 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave9 2023-09-22T13:23:32.169 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CREATE_BUCKET_MULTI (71 ms) 2023-09-22T13:23:32.169 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.DELETE_BUCKET_MULTI 2023-09-22T13:23:32.466 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.DELETE_BUCKET_MULTI (297 ms) 2023-09-22T13:23:32.466 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CLEANUP 2023-09-22T13:23:32.466 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-09-22T13:23:32.466 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.UMOUNT 2023-09-22T13:23:32.466 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-09-22T13:23:32.467 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-09-22T13:23:32.467 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:32.465+0000 7f6892e6e540 -1 shutting down 2023-09-22T13:23:32.467 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:32.465+0000 7f67a9ffb700 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 148659, si_uid : 1000, si_addr0x3e8000244b3, si_status0 2023-09-22T13:23:32.489 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SHUTDOWN (24 ms) 2023-09-22T13:23:32.489 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 9 tests from LibRGW (483 ms total) 2023-09-22T13:23:32.490 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:23:32.490 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:23:32.490 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 9 tests from 1 test suite ran. (483 ms total) 2023-09-22T13:23:32.490 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 9 tests. 2023-09-22T13:23:32.491 INFO:tasks.workunit.client.0.smithi096.stdout:phase 2.2 2023-09-22T13:23:32.507 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 9 tests from 1 test suite. 2023-09-22T13:23:32.507 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:23:32.507 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 9 tests from LibRGW 2023-09-22T13:23:32.507 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.INIT 2023-09-22T13:23:32.515 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:23:32.515 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:23:32.606 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.INIT (99 ms) 2023-09-22T13:23:32.606 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MOUNT 2023-09-22T13:23:32.607 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MOUNT (2 ms) 2023-09-22T13:23:32.607 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-09-22T13:23:32.607 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CREATE_BUCKET (0 ms) 2023-09-22T13:23:32.607 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-09-22T13:23:32.640 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.DELETE_BUCKET (32 ms) 2023-09-22T13:23:32.640 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CREATE_BUCKET_MULTI 2023-09-22T13:23:32.646 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave0 2023-09-22T13:23:32.653 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave1 2023-09-22T13:23:32.660 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave2 2023-09-22T13:23:32.666 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave3 2023-09-22T13:23:32.673 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave4 2023-09-22T13:23:32.679 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave5 2023-09-22T13:23:32.686 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave6 2023-09-22T13:23:32.693 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave7 2023-09-22T13:23:32.700 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave8 2023-09-22T13:23:32.706 INFO:tasks.workunit.client.0.smithi096.stdout:created: sorrydave9 2023-09-22T13:23:32.706 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CREATE_BUCKET_MULTI (67 ms) 2023-09-22T13:23:32.707 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.DELETE_BUCKET_MULTI 2023-09-22T13:23:33.010 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.DELETE_BUCKET_MULTI (303 ms) 2023-09-22T13:23:33.010 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CLEANUP 2023-09-22T13:23:33.010 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-09-22T13:23:33.010 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.UMOUNT 2023-09-22T13:23:33.010 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-09-22T13:23:33.010 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-09-22T13:23:33.010 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:33.009+0000 7f855caab540 -1 shutting down 2023-09-22T13:23:33.011 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:33.009+0000 7f8470ff9700 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 148743, si_uid : 1000, si_addr0x3e800024507, si_status0 2023-09-22T13:23:33.107 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SHUTDOWN (97 ms) 2023-09-22T13:23:33.107 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 9 tests from LibRGW (600 ms total) 2023-09-22T13:23:33.107 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:23:33.107 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:23:33.107 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 9 tests from 1 test suite ran. (600 ms total) 2023-09-22T13:23:33.107 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 9 tests. 2023-09-22T13:23:33.109 INFO:tasks.workunit.client.0.smithi096.stdout:phase 3.1 2023-09-22T13:23:33.854 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 13 tests from 1 test suite. 2023-09-22T13:23:33.854 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:23:33.854 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 13 tests from LibRGW 2023-09-22T13:23:33.854 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.INIT 2023-09-22T13:23:33.863 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:23:33.863 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:23:34.352 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.INIT (497 ms) 2023-09-22T13:23:34.352 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MOUNT 2023-09-22T13:23:34.353 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MOUNT (2 ms) 2023-09-22T13:23:34.353 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-09-22T13:23:34.361 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CREATE_BUCKET (8 ms) 2023-09-22T13:23:34.361 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.LOOKUP_BUCKET 2023-09-22T13:23:34.364 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.LOOKUP_BUCKET (3 ms) 2023-09-22T13:23:34.364 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.LOOKUP_OBJECT 2023-09-22T13:23:34.366 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.LOOKUP_OBJECT (2 ms) 2023-09-22T13:23:34.366 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.LARGE1 2023-09-22T13:23:34.803 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.LARGE1 (436 ms) 2023-09-22T13:23:34.803 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.LARGE2 2023-09-22T13:23:34.837 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.LARGE2 (35 ms) 2023-09-22T13:23:34.838 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.STAT_OBJECT 2023-09-22T13:23:34.838 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.STAT_OBJECT (0 ms) 2023-09-22T13:23:34.838 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.DELETE_OBJECT 2023-09-22T13:23:34.838 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.DELETE_OBJECT (0 ms) 2023-09-22T13:23:34.838 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-09-22T13:23:34.838 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.DELETE_BUCKET (0 ms) 2023-09-22T13:23:34.838 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CLEANUP 2023-09-22T13:23:34.838 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-09-22T13:23:34.838 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.UMOUNT 2023-09-22T13:23:34.838 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-09-22T13:23:34.839 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-09-22T13:23:34.839 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:34.837+0000 7f148697e540 -1 shutting down 2023-09-22T13:23:34.839 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:34.837+0000 7f1387fff700 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 148827, si_uid : 1000, si_addr0x3e80002455b, si_status0 2023-09-22T13:23:34.852 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SHUTDOWN (15 ms) 2023-09-22T13:23:34.853 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 13 tests from LibRGW (998 ms total) 2023-09-22T13:23:34.853 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:23:34.853 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:23:34.853 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 13 tests from 1 test suite ran. (998 ms total) 2023-09-22T13:23:34.853 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 13 tests. 2023-09-22T13:23:34.856 INFO:tasks.workunit.client.0.smithi096.stdout:phase 3.2 2023-09-22T13:23:35.612 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 13 tests from 1 test suite. 2023-09-22T13:23:35.612 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:23:35.612 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 13 tests from LibRGW 2023-09-22T13:23:35.612 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.INIT 2023-09-22T13:23:35.619 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:23:35.619 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:23:35.912 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.INIT (300 ms) 2023-09-22T13:23:35.912 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MOUNT 2023-09-22T13:23:35.913 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MOUNT (2 ms) 2023-09-22T13:23:35.913 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-09-22T13:23:35.914 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CREATE_BUCKET (0 ms) 2023-09-22T13:23:35.914 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.LOOKUP_BUCKET 2023-09-22T13:23:35.918 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.LOOKUP_BUCKET (5 ms) 2023-09-22T13:23:35.918 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.LOOKUP_OBJECT 2023-09-22T13:23:35.922 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.LOOKUP_OBJECT (4 ms) 2023-09-22T13:23:35.922 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.LARGE1 2023-09-22T13:23:35.985 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.LARGE1 (63 ms) 2023-09-22T13:23:35.986 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.LARGE2 2023-09-22T13:23:35.986 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.LARGE2 (0 ms) 2023-09-22T13:23:35.986 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.STAT_OBJECT 2023-09-22T13:23:35.986 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.STAT_OBJECT (0 ms) 2023-09-22T13:23:35.986 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.DELETE_OBJECT 2023-09-22T13:23:35.990 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.DELETE_OBJECT (4 ms) 2023-09-22T13:23:35.990 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-09-22T13:23:36.019 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.DELETE_BUCKET (30 ms) 2023-09-22T13:23:36.019 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CLEANUP 2023-09-22T13:23:36.019 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-09-22T13:23:36.019 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.UMOUNT 2023-09-22T13:23:36.020 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-09-22T13:23:36.020 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-09-22T13:23:36.020 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:36.019+0000 7fa0d5b1a540 -1 shutting down 2023-09-22T13:23:36.020 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:36.019+0000 7f9fdb7fe700 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 148911, si_uid : 1000, si_addr0x3e8000245af, si_status0 2023-09-22T13:23:36.113 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SHUTDOWN (93 ms) 2023-09-22T13:23:36.113 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 13 tests from LibRGW (501 ms total) 2023-09-22T13:23:36.113 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:23:36.113 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:23:36.113 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 13 tests from 1 test suite ran. (501 ms total) 2023-09-22T13:23:36.113 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 13 tests. 2023-09-22T13:23:36.115 INFO:tasks.workunit.client.0.smithi096.stdout:phase 4.1 2023-09-22T13:23:36.131 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 11 tests from 1 test suite. 2023-09-22T13:23:36.131 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:23:36.131 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 11 tests from LibRGW 2023-09-22T13:23:36.131 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.TVAR 2023-09-22T13:23:36.131 INFO:tasks.workunit.client.0.smithi096.stdout:read i1: 64001 2023-09-22T13:23:36.131 INFO:tasks.workunit.client.0.smithi096.stdout:read s1: blunderbuss 2023-09-22T13:23:36.131 INFO:tasks.workunit.client.0.smithi096.stdout:read s3: [ OK ] LibRGW.TVAR (0 ms) 2023-09-22T13:23:36.131 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.INIT 2023-09-22T13:23:36.142 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:23:36.142 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:23:36.229 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.INIT (97 ms) 2023-09-22T13:23:36.229 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MOUNT 2023-09-22T13:23:36.230 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MOUNT (2 ms) 2023-09-22T13:23:36.230 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2023-09-22T13:23:36.246 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (16 ms) 2023-09-22T13:23:36.246 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2023-09-22T13:23:36.696 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_0 2023-09-22T13:23:36.696 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_1 2023-09-22T13:23:36.696 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_2 2023-09-22T13:23:36.696 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_3 2023-09-22T13:23:36.697 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_4 2023-09-22T13:23:36.697 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_5 2023-09-22T13:23:36.697 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_6 2023-09-22T13:23:36.697 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_7 2023-09-22T13:23:36.697 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_8 2023-09-22T13:23:36.697 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_9 2023-09-22T13:23:36.697 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_10 2023-09-22T13:23:36.697 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_11 2023-09-22T13:23:36.697 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_12 2023-09-22T13:23:36.697 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_13 2023-09-22T13:23:36.697 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_14 2023-09-22T13:23:36.697 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_15 2023-09-22T13:23:36.698 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_16 2023-09-22T13:23:36.698 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_17 2023-09-22T13:23:36.698 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_18 2023-09-22T13:23:36.698 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_19 2023-09-22T13:23:36.698 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_20 2023-09-22T13:23:36.698 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_21 2023-09-22T13:23:36.698 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_22 2023-09-22T13:23:36.698 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_23 2023-09-22T13:23:36.698 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_24 2023-09-22T13:23:36.698 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_25 2023-09-22T13:23:36.698 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_26 2023-09-22T13:23:36.698 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_27 2023-09-22T13:23:36.698 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_28 2023-09-22T13:23:36.699 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_29 2023-09-22T13:23:36.699 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_30 2023-09-22T13:23:36.699 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_31 2023-09-22T13:23:36.699 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_32 2023-09-22T13:23:36.699 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_33 2023-09-22T13:23:36.699 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_34 2023-09-22T13:23:36.699 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_35 2023-09-22T13:23:36.699 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_36 2023-09-22T13:23:36.699 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_37 2023-09-22T13:23:36.699 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_38 2023-09-22T13:23:36.699 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_39 2023-09-22T13:23:36.699 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_40 2023-09-22T13:23:36.699 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_41 2023-09-22T13:23:36.700 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_42 2023-09-22T13:23:36.700 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_43 2023-09-22T13:23:36.700 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_44 2023-09-22T13:23:36.700 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_45 2023-09-22T13:23:36.700 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_46 2023-09-22T13:23:36.700 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_47 2023-09-22T13:23:36.700 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_48 2023-09-22T13:23:36.700 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_49 2023-09-22T13:23:36.700 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_50 2023-09-22T13:23:36.700 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_51 2023-09-22T13:23:36.700 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_52 2023-09-22T13:23:36.700 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_53 2023-09-22T13:23:36.700 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_54 2023-09-22T13:23:36.701 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_55 2023-09-22T13:23:36.701 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_56 2023-09-22T13:23:36.701 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_57 2023-09-22T13:23:36.701 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_58 2023-09-22T13:23:36.701 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_59 2023-09-22T13:23:36.701 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_60 2023-09-22T13:23:36.701 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_61 2023-09-22T13:23:36.701 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_62 2023-09-22T13:23:36.701 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_63 2023-09-22T13:23:36.701 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_64 2023-09-22T13:23:36.701 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_65 2023-09-22T13:23:36.701 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_66 2023-09-22T13:23:36.701 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_67 2023-09-22T13:23:36.702 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_68 2023-09-22T13:23:36.702 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_69 2023-09-22T13:23:36.702 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_70 2023-09-22T13:23:36.702 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_71 2023-09-22T13:23:36.702 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_72 2023-09-22T13:23:36.702 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_73 2023-09-22T13:23:36.702 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_74 2023-09-22T13:23:36.702 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_75 2023-09-22T13:23:36.702 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_76 2023-09-22T13:23:36.702 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_77 2023-09-22T13:23:36.702 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_78 2023-09-22T13:23:36.702 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_79 2023-09-22T13:23:36.702 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_80 2023-09-22T13:23:36.703 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_81 2023-09-22T13:23:36.703 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_82 2023-09-22T13:23:36.703 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_83 2023-09-22T13:23:36.703 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_84 2023-09-22T13:23:36.703 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_85 2023-09-22T13:23:36.703 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_86 2023-09-22T13:23:36.703 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_87 2023-09-22T13:23:36.703 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_88 2023-09-22T13:23:36.703 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_89 2023-09-22T13:23:36.703 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_90 2023-09-22T13:23:36.703 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_91 2023-09-22T13:23:36.703 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_92 2023-09-22T13:23:36.703 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_93 2023-09-22T13:23:36.704 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_94 2023-09-22T13:23:36.704 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_95 2023-09-22T13:23:36.704 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_96 2023-09-22T13:23:36.704 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_97 2023-09-22T13:23:36.704 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_98 2023-09-22T13:23:36.704 INFO:tasks.workunit.client.0.smithi096.stdout:created: dmarker:f_99 2023-09-22T13:23:36.704 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (450 ms) 2023-09-22T13:23:36.704 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2023-09-22T13:23:36.706 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=. flags=4 2023-09-22T13:23:36.706 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=.. flags=4 2023-09-22T13:23:36.706 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_0 flags=8 2023-09-22T13:23:36.706 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_1 flags=8 2023-09-22T13:23:36.706 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_10 flags=8 2023-09-22T13:23:36.706 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_11 flags=8 2023-09-22T13:23:36.706 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_12 flags=8 2023-09-22T13:23:36.706 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_13 flags=8 2023-09-22T13:23:36.706 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_14 flags=8 2023-09-22T13:23:36.706 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_15 flags=8 2023-09-22T13:23:36.706 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_16 flags=8 2023-09-22T13:23:36.707 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_17 flags=8 2023-09-22T13:23:36.707 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_18 flags=8 2023-09-22T13:23:36.707 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_19 flags=8 2023-09-22T13:23:36.707 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_2 flags=8 2023-09-22T13:23:36.707 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_20 flags=8 2023-09-22T13:23:36.707 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_21 flags=8 2023-09-22T13:23:36.707 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_22 flags=8 2023-09-22T13:23:36.707 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_23 flags=8 2023-09-22T13:23:36.707 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_24 flags=8 2023-09-22T13:23:36.707 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_25 flags=8 2023-09-22T13:23:36.707 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_26 flags=8 2023-09-22T13:23:36.707 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_27 flags=8 2023-09-22T13:23:36.707 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_28 flags=8 2023-09-22T13:23:36.708 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_29 flags=8 2023-09-22T13:23:36.708 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_3 flags=8 2023-09-22T13:23:36.708 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_30 flags=8 2023-09-22T13:23:36.708 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_31 flags=8 2023-09-22T13:23:36.708 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_32 flags=8 2023-09-22T13:23:36.708 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_33 flags=8 2023-09-22T13:23:36.708 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_34 flags=8 2023-09-22T13:23:36.708 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_35 flags=8 2023-09-22T13:23:36.708 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_36 flags=8 2023-09-22T13:23:36.708 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_37 flags=8 2023-09-22T13:23:36.708 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_38 flags=8 2023-09-22T13:23:36.708 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_39 flags=8 2023-09-22T13:23:36.708 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_4 flags=8 2023-09-22T13:23:36.709 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_40 flags=8 2023-09-22T13:23:36.709 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_41 flags=8 2023-09-22T13:23:36.709 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_42 flags=8 2023-09-22T13:23:36.709 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_43 flags=8 2023-09-22T13:23:36.709 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_44 flags=8 2023-09-22T13:23:36.709 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_45 flags=8 2023-09-22T13:23:36.709 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_46 flags=8 2023-09-22T13:23:36.709 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_47 flags=8 2023-09-22T13:23:36.709 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_48 flags=8 2023-09-22T13:23:36.709 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_49 flags=8 2023-09-22T13:23:36.710 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_5 flags=8 2023-09-22T13:23:36.710 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_50 flags=8 2023-09-22T13:23:36.710 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_51 flags=8 2023-09-22T13:23:36.710 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_52 flags=8 2023-09-22T13:23:36.710 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_53 flags=8 2023-09-22T13:23:36.710 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_54 flags=8 2023-09-22T13:23:36.710 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_55 flags=8 2023-09-22T13:23:36.710 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_56 flags=8 2023-09-22T13:23:36.710 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_57 flags=8 2023-09-22T13:23:36.710 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_58 flags=8 2023-09-22T13:23:36.710 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_59 flags=8 2023-09-22T13:23:36.710 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_6 flags=8 2023-09-22T13:23:36.711 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_60 flags=8 2023-09-22T13:23:36.711 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_61 flags=8 2023-09-22T13:23:36.711 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_62 flags=8 2023-09-22T13:23:36.711 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_63 flags=8 2023-09-22T13:23:36.711 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_64 flags=8 2023-09-22T13:23:36.711 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_65 flags=8 2023-09-22T13:23:36.711 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_66 flags=8 2023-09-22T13:23:36.711 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_67 flags=8 2023-09-22T13:23:36.711 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_68 flags=8 2023-09-22T13:23:36.711 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_69 flags=8 2023-09-22T13:23:36.711 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_7 flags=8 2023-09-22T13:23:36.711 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_70 flags=8 2023-09-22T13:23:36.711 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_71 flags=8 2023-09-22T13:23:36.712 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_72 flags=8 2023-09-22T13:23:36.712 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_73 flags=8 2023-09-22T13:23:36.712 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_74 flags=8 2023-09-22T13:23:36.712 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_75 flags=8 2023-09-22T13:23:36.712 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_76 flags=8 2023-09-22T13:23:36.712 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_77 flags=8 2023-09-22T13:23:36.712 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_78 flags=8 2023-09-22T13:23:36.712 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_79 flags=8 2023-09-22T13:23:36.712 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_8 flags=8 2023-09-22T13:23:36.712 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_80 flags=8 2023-09-22T13:23:36.712 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_81 flags=8 2023-09-22T13:23:36.712 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_82 flags=8 2023-09-22T13:23:36.712 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_83 flags=8 2023-09-22T13:23:36.713 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_84 flags=8 2023-09-22T13:23:36.713 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_85 flags=8 2023-09-22T13:23:36.713 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_86 flags=8 2023-09-22T13:23:36.713 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_87 flags=8 2023-09-22T13:23:36.713 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_88 flags=8 2023-09-22T13:23:36.713 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_89 flags=8 2023-09-22T13:23:36.713 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_9 flags=8 2023-09-22T13:23:36.713 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_90 flags=8 2023-09-22T13:23:36.713 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_91 flags=8 2023-09-22T13:23:36.713 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_92 flags=8 2023-09-22T13:23:36.714 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_93 flags=8 2023-09-22T13:23:36.714 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_94 flags=8 2023-09-22T13:23:36.714 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_95 flags=8 2023-09-22T13:23:36.714 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_96 flags=8 2023-09-22T13:23:36.714 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_97 flags=8 2023-09-22T13:23:36.714 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_98 flags=8 2023-09-22T13:23:36.714 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_99 flags=8 2023-09-22T13:23:36.714 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_READDIR (2 ms) 2023-09-22T13:23:36.714 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER2_READDIR 2023-09-22T13:23:36.714 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_0 flags=8 2023-09-22T13:23:36.714 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_1 flags=8 2023-09-22T13:23:36.714 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_10 flags=8 2023-09-22T13:23:36.715 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_11 flags=8 2023-09-22T13:23:36.715 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_12 flags=8 2023-09-22T13:23:36.715 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_13 flags=8 2023-09-22T13:23:36.715 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_14 flags=8 2023-09-22T13:23:36.715 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_15 flags=8 2023-09-22T13:23:36.715 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_16 flags=8 2023-09-22T13:23:36.715 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_17 flags=8 2023-09-22T13:23:36.715 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_18 flags=8 2023-09-22T13:23:36.715 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_19 flags=8 2023-09-22T13:23:36.715 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_2 flags=8 2023-09-22T13:23:36.715 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_20 flags=8 2023-09-22T13:23:36.715 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_21 flags=8 2023-09-22T13:23:36.715 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_22 flags=8 2023-09-22T13:23:36.716 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_23 flags=8 2023-09-22T13:23:36.716 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_24 flags=8 2023-09-22T13:23:36.716 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_25 flags=8 2023-09-22T13:23:36.716 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_26 flags=8 2023-09-22T13:23:36.716 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_27 flags=8 2023-09-22T13:23:36.716 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_28 flags=8 2023-09-22T13:23:36.716 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_29 flags=8 2023-09-22T13:23:36.716 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_3 flags=8 2023-09-22T13:23:36.716 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_30 flags=8 2023-09-22T13:23:36.716 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_31 flags=8 2023-09-22T13:23:36.716 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_32 flags=8 2023-09-22T13:23:36.716 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_33 flags=8 2023-09-22T13:23:36.716 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_34 flags=8 2023-09-22T13:23:36.717 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_35 flags=8 2023-09-22T13:23:36.717 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_36 flags=8 2023-09-22T13:23:36.717 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_37 flags=8 2023-09-22T13:23:36.717 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_38 flags=8 2023-09-22T13:23:36.717 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_39 flags=8 2023-09-22T13:23:36.717 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_4 flags=8 2023-09-22T13:23:36.717 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_40 flags=8 2023-09-22T13:23:36.717 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_41 flags=8 2023-09-22T13:23:36.717 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_42 flags=8 2023-09-22T13:23:36.717 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_43 flags=8 2023-09-22T13:23:36.717 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_44 flags=8 2023-09-22T13:23:36.717 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_45 flags=8 2023-09-22T13:23:36.719 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_46 flags=8 2023-09-22T13:23:36.719 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_47 flags=8 2023-09-22T13:23:36.719 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_48 flags=8 2023-09-22T13:23:36.719 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_49 flags=8 2023-09-22T13:23:36.719 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_5 flags=8 2023-09-22T13:23:36.719 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_50 flags=8 2023-09-22T13:23:36.719 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_51 flags=8 2023-09-22T13:23:36.719 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_52 flags=8 2023-09-22T13:23:36.719 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_53 flags=8 2023-09-22T13:23:36.719 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_54 flags=8 2023-09-22T13:23:36.719 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_55 flags=8 2023-09-22T13:23:36.719 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_56 flags=8 2023-09-22T13:23:36.720 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_57 flags=8 2023-09-22T13:23:36.720 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_58 flags=8 2023-09-22T13:23:36.720 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_59 flags=8 2023-09-22T13:23:36.720 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_6 flags=8 2023-09-22T13:23:36.720 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_60 flags=8 2023-09-22T13:23:36.720 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_61 flags=8 2023-09-22T13:23:36.720 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_62 flags=8 2023-09-22T13:23:36.720 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_63 flags=8 2023-09-22T13:23:36.720 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_64 flags=8 2023-09-22T13:23:36.720 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_65 flags=8 2023-09-22T13:23:36.720 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_66 flags=8 2023-09-22T13:23:36.721 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_67 flags=8 2023-09-22T13:23:36.721 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_68 flags=8 2023-09-22T13:23:36.721 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_69 flags=8 2023-09-22T13:23:36.721 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_7 flags=8 2023-09-22T13:23:36.721 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_70 flags=8 2023-09-22T13:23:36.721 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_71 flags=8 2023-09-22T13:23:36.721 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_72 flags=8 2023-09-22T13:23:36.721 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_73 flags=8 2023-09-22T13:23:36.721 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_74 flags=8 2023-09-22T13:23:36.721 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_75 flags=8 2023-09-22T13:23:36.721 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_76 flags=8 2023-09-22T13:23:36.721 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_77 flags=8 2023-09-22T13:23:36.722 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_78 flags=8 2023-09-22T13:23:36.722 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_79 flags=8 2023-09-22T13:23:36.722 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_8 flags=8 2023-09-22T13:23:36.722 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_80 flags=8 2023-09-22T13:23:36.722 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_81 flags=8 2023-09-22T13:23:36.722 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_82 flags=8 2023-09-22T13:23:36.722 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_83 flags=8 2023-09-22T13:23:36.722 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_84 flags=8 2023-09-22T13:23:36.722 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_85 flags=8 2023-09-22T13:23:36.722 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_86 flags=8 2023-09-22T13:23:36.722 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_87 flags=8 2023-09-22T13:23:36.722 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_88 flags=8 2023-09-22T13:23:36.723 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_89 flags=8 2023-09-22T13:23:36.723 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_9 flags=8 2023-09-22T13:23:36.723 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_90 flags=8 2023-09-22T13:23:36.723 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_91 flags=8 2023-09-22T13:23:36.723 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_92 flags=8 2023-09-22T13:23:36.723 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_93 flags=8 2023-09-22T13:23:36.723 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_94 flags=8 2023-09-22T13:23:36.723 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_95 flags=8 2023-09-22T13:23:36.723 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_96 flags=8 2023-09-22T13:23:36.723 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_97 flags=8 2023-09-22T13:23:36.723 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_98 flags=8 2023-09-22T13:23:36.723 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_99 flags=8 2023-09-22T13:23:36.724 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER2_READDIR (2 ms) 2023-09-22T13:23:36.724 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2023-09-22T13:23:36.724 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2023-09-22T13:23:36.724 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CLEANUP 2023-09-22T13:23:36.724 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-09-22T13:23:36.724 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.UMOUNT 2023-09-22T13:23:36.724 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-09-22T13:23:36.724 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-09-22T13:23:36.724 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:36.700+0000 7f993a098540 -1 shutting down 2023-09-22T13:23:36.724 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:36.700+0000 7f984dffb700 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 148995, si_uid : 1000, si_addr0x3e800024603, si_status0 2023-09-22T13:23:36.730 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SHUTDOWN (30 ms) 2023-09-22T13:23:36.730 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 11 tests from LibRGW (599 ms total) 2023-09-22T13:23:36.730 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:23:36.731 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:23:36.731 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 11 tests from 1 test suite ran. (600 ms total) 2023-09-22T13:23:36.731 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 11 tests. 2023-09-22T13:23:36.732 INFO:tasks.workunit.client.0.smithi096.stdout:phase 4.2 2023-09-22T13:23:36.748 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 11 tests from 1 test suite. 2023-09-22T13:23:36.748 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:23:36.748 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 11 tests from LibRGW 2023-09-22T13:23:36.748 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.TVAR 2023-09-22T13:23:36.748 INFO:tasks.workunit.client.0.smithi096.stdout:read i1: 64001 2023-09-22T13:23:36.748 INFO:tasks.workunit.client.0.smithi096.stdout:read s1: blunderbuss 2023-09-22T13:23:36.748 INFO:tasks.workunit.client.0.smithi096.stdout:read s3: [ OK ] LibRGW.TVAR (0 ms) 2023-09-22T13:23:36.748 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.INIT 2023-09-22T13:23:36.756 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:23:36.756 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:23:36.832 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.INIT (85 ms) 2023-09-22T13:23:36.833 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MOUNT 2023-09-22T13:23:36.833 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MOUNT (1 ms) 2023-09-22T13:23:36.833 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2023-09-22T13:23:36.843 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (10 ms) 2023-09-22T13:23:36.843 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2023-09-22T13:23:36.843 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (0 ms) 2023-09-22T13:23:36.843 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2023-09-22T13:23:36.844 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_READDIR (0 ms) 2023-09-22T13:23:36.844 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER2_READDIR 2023-09-22T13:23:36.845 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_0 flags=8 2023-09-22T13:23:36.846 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_1 flags=8 2023-09-22T13:23:36.846 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_10 flags=8 2023-09-22T13:23:36.846 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_11 flags=8 2023-09-22T13:23:36.846 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_12 flags=8 2023-09-22T13:23:36.846 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_13 flags=8 2023-09-22T13:23:36.846 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_14 flags=8 2023-09-22T13:23:36.846 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_15 flags=8 2023-09-22T13:23:36.846 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_16 flags=8 2023-09-22T13:23:36.846 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_17 flags=8 2023-09-22T13:23:36.846 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_18 flags=8 2023-09-22T13:23:36.847 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_19 flags=8 2023-09-22T13:23:36.847 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_2 flags=8 2023-09-22T13:23:36.847 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_20 flags=8 2023-09-22T13:23:36.847 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_21 flags=8 2023-09-22T13:23:36.847 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_22 flags=8 2023-09-22T13:23:36.847 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_23 flags=8 2023-09-22T13:23:36.847 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_24 flags=8 2023-09-22T13:23:36.847 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_25 flags=8 2023-09-22T13:23:36.847 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_26 flags=8 2023-09-22T13:23:36.847 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_27 flags=8 2023-09-22T13:23:36.847 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_28 flags=8 2023-09-22T13:23:36.848 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_29 flags=8 2023-09-22T13:23:36.848 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_3 flags=8 2023-09-22T13:23:36.848 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_30 flags=8 2023-09-22T13:23:36.848 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_31 flags=8 2023-09-22T13:23:36.848 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_32 flags=8 2023-09-22T13:23:36.848 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_33 flags=8 2023-09-22T13:23:36.848 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_34 flags=8 2023-09-22T13:23:36.848 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_35 flags=8 2023-09-22T13:23:36.848 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_36 flags=8 2023-09-22T13:23:36.848 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_37 flags=8 2023-09-22T13:23:36.848 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_38 flags=8 2023-09-22T13:23:36.848 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_39 flags=8 2023-09-22T13:23:36.849 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_4 flags=8 2023-09-22T13:23:36.849 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_40 flags=8 2023-09-22T13:23:36.849 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_41 flags=8 2023-09-22T13:23:36.849 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_42 flags=8 2023-09-22T13:23:36.849 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_43 flags=8 2023-09-22T13:23:36.849 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_44 flags=8 2023-09-22T13:23:36.849 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_45 flags=8 2023-09-22T13:23:36.849 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_46 flags=8 2023-09-22T13:23:36.849 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_47 flags=8 2023-09-22T13:23:36.849 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_48 flags=8 2023-09-22T13:23:36.849 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_49 flags=8 2023-09-22T13:23:36.850 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_5 flags=8 2023-09-22T13:23:36.850 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_50 flags=8 2023-09-22T13:23:36.850 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_51 flags=8 2023-09-22T13:23:36.850 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_52 flags=8 2023-09-22T13:23:36.850 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_53 flags=8 2023-09-22T13:23:36.850 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_54 flags=8 2023-09-22T13:23:36.850 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_55 flags=8 2023-09-22T13:23:36.850 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_56 flags=8 2023-09-22T13:23:36.850 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_57 flags=8 2023-09-22T13:23:36.850 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_58 flags=8 2023-09-22T13:23:36.850 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_59 flags=8 2023-09-22T13:23:36.851 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_6 flags=8 2023-09-22T13:23:36.851 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_60 flags=8 2023-09-22T13:23:36.851 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_61 flags=8 2023-09-22T13:23:36.851 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_62 flags=8 2023-09-22T13:23:36.851 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_63 flags=8 2023-09-22T13:23:36.851 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_64 flags=8 2023-09-22T13:23:36.851 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_65 flags=8 2023-09-22T13:23:36.851 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_66 flags=8 2023-09-22T13:23:36.851 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_67 flags=8 2023-09-22T13:23:36.851 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_68 flags=8 2023-09-22T13:23:36.851 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_69 flags=8 2023-09-22T13:23:36.851 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_7 flags=8 2023-09-22T13:23:36.852 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_70 flags=8 2023-09-22T13:23:36.852 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_71 flags=8 2023-09-22T13:23:36.852 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_72 flags=8 2023-09-22T13:23:36.852 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_73 flags=8 2023-09-22T13:23:36.852 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_74 flags=8 2023-09-22T13:23:36.852 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_75 flags=8 2023-09-22T13:23:36.852 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_76 flags=8 2023-09-22T13:23:36.852 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_77 flags=8 2023-09-22T13:23:36.852 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_78 flags=8 2023-09-22T13:23:36.852 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_79 flags=8 2023-09-22T13:23:36.852 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_8 flags=8 2023-09-22T13:23:36.852 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_80 flags=8 2023-09-22T13:23:36.853 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_81 flags=8 2023-09-22T13:23:36.853 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_82 flags=8 2023-09-22T13:23:36.853 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_83 flags=8 2023-09-22T13:23:36.853 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_84 flags=8 2023-09-22T13:23:36.853 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_85 flags=8 2023-09-22T13:23:36.853 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_86 flags=8 2023-09-22T13:23:36.853 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_87 flags=8 2023-09-22T13:23:36.853 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_88 flags=8 2023-09-22T13:23:36.853 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_89 flags=8 2023-09-22T13:23:36.853 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_9 flags=8 2023-09-22T13:23:36.853 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_90 flags=8 2023-09-22T13:23:36.853 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_91 flags=8 2023-09-22T13:23:36.854 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_92 flags=8 2023-09-22T13:23:36.854 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_93 flags=8 2023-09-22T13:23:36.854 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_94 flags=8 2023-09-22T13:23:36.854 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_95 flags=8 2023-09-22T13:23:36.854 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_96 flags=8 2023-09-22T13:23:36.854 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_97 flags=8 2023-09-22T13:23:36.854 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_98 flags=8 2023-09-22T13:23:36.854 INFO:tasks.workunit.client.0.smithi096.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_99 flags=8 2023-09-22T13:23:36.854 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER2_READDIR (2 ms) 2023-09-22T13:23:36.854 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2023-09-22T13:23:36.854 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2023-09-22T13:23:36.854 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CLEANUP 2023-09-22T13:23:36.855 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-09-22T13:23:36.855 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.UMOUNT 2023-09-22T13:23:36.855 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-09-22T13:23:36.855 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-09-22T13:23:36.855 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:36.845+0000 7f0db2ab3540 -1 shutting down 2023-09-22T13:23:36.855 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:36.845+0000 7f0cc9ffb700 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 149079, si_uid : 1000, si_addr0x3e800024657, si_status0 2023-09-22T13:23:36.933 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SHUTDOWN (87 ms) 2023-09-22T13:23:36.937 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 11 tests from LibRGW (185 ms total) 2023-09-22T13:23:36.937 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:23:36.937 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:23:36.938 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 11 tests from 1 test suite ran. (185 ms total) 2023-09-22T13:23:36.938 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 11 tests. 2023-09-22T13:23:36.938 INFO:tasks.workunit.client.0.smithi096.stdout:phase 5.1 2023-09-22T13:23:36.974 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 19 tests from 1 test suite. 2023-09-22T13:23:36.974 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:23:36.974 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 19 tests from LibRGW 2023-09-22T13:23:36.974 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.INIT 2023-09-22T13:23:36.983 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:23:36.983 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:23:37.070 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.INIT (96 ms) 2023-09-22T13:23:37.070 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MOUNT 2023-09-22T13:23:37.071 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MOUNT (2 ms) 2023-09-22T13:23:37.071 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-09-22T13:23:37.078 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CREATE_BUCKET (7 ms) 2023-09-22T13:23:37.078 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.LOOKUP_BUCKET 2023-09-22T13:23:37.081 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.LOOKUP_BUCKET (3 ms) 2023-09-22T13:23:37.081 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.LIST_OBJECTS 2023-09-22T13:23:37.081 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.LIST_OBJECTS (0 ms) 2023-09-22T13:23:37.081 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.LOOKUP_OBJECT 2023-09-22T13:23:37.083 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.LOOKUP_OBJECT (2 ms) 2023-09-22T13:23:37.083 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.OBJ_OPEN 2023-09-22T13:23:37.083 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.OBJ_OPEN (0 ms) 2023-09-22T13:23:37.083 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.PUT_OBJECT 2023-09-22T13:23:37.086 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.PUT_OBJECT (2 ms) 2023-09-22T13:23:37.086 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.GET_OBJECT 2023-09-22T13:23:37.087 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.GET_OBJECT (0 ms) 2023-09-22T13:23:37.087 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.STAT_OBJECT 2023-09-22T13:23:37.087 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.STAT_OBJECT (0 ms) 2023-09-22T13:23:37.087 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.WRITE_READ_VERIFY 2023-09-22T13:23:37.087 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.WRITE_READ_VERIFY (0 ms) 2023-09-22T13:23:37.087 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.WRITEV 2023-09-22T13:23:37.087 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.WRITEV (0 ms) 2023-09-22T13:23:37.087 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.READV 2023-09-22T13:23:37.087 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.READV (0 ms) 2023-09-22T13:23:37.087 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.READV_AFTER_WRITEV 2023-09-22T13:23:37.088 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.READV_AFTER_WRITEV (0 ms) 2023-09-22T13:23:37.088 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.DELETE_OBJECT 2023-09-22T13:23:37.088 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.DELETE_OBJECT (0 ms) 2023-09-22T13:23:37.088 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-09-22T13:23:37.088 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.DELETE_BUCKET (0 ms) 2023-09-22T13:23:37.088 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CLEANUP 2023-09-22T13:23:37.088 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-09-22T13:23:37.088 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.UMOUNT 2023-09-22T13:23:37.088 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-09-22T13:23:37.088 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-09-22T13:23:37.089 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:37.086+0000 7f40a7c8f540 -1 shutting down 2023-09-22T13:23:37.089 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:37.086+0000 7f3faf7fe700 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 149163, si_uid : 1000, si_addr0x3e8000246ab, si_status0 2023-09-22T13:23:37.170 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SHUTDOWN (84 ms) 2023-09-22T13:23:37.171 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 19 tests from LibRGW (197 ms total) 2023-09-22T13:23:37.171 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:23:37.171 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:23:37.171 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 19 tests from 1 test suite ran. (197 ms total) 2023-09-22T13:23:37.171 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 19 tests. 2023-09-22T13:23:37.172 INFO:tasks.workunit.client.0.smithi096.stdout:phase 5.2 2023-09-22T13:23:37.211 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] Running 19 tests from 1 test suite. 2023-09-22T13:23:37.211 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment set-up. 2023-09-22T13:23:37.211 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 19 tests from LibRGW 2023-09-22T13:23:37.211 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.INIT 2023-09-22T13:23:37.217 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:23:37.218 INFO:tasks.workunit.client.0.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:23:37.305 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.INIT (93 ms) 2023-09-22T13:23:37.305 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.MOUNT 2023-09-22T13:23:37.306 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.MOUNT (2 ms) 2023-09-22T13:23:37.306 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-09-22T13:23:37.306 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CREATE_BUCKET (0 ms) 2023-09-22T13:23:37.306 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.LOOKUP_BUCKET 2023-09-22T13:23:37.312 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.LOOKUP_BUCKET (5 ms) 2023-09-22T13:23:37.312 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.LIST_OBJECTS 2023-09-22T13:23:37.312 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.LIST_OBJECTS (0 ms) 2023-09-22T13:23:37.312 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.LOOKUP_OBJECT 2023-09-22T13:23:37.312 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.LOOKUP_OBJECT (0 ms) 2023-09-22T13:23:37.313 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.OBJ_OPEN 2023-09-22T13:23:37.313 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.OBJ_OPEN (0 ms) 2023-09-22T13:23:37.313 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.PUT_OBJECT 2023-09-22T13:23:37.313 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.PUT_OBJECT (0 ms) 2023-09-22T13:23:37.313 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.GET_OBJECT 2023-09-22T13:23:37.313 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.GET_OBJECT (0 ms) 2023-09-22T13:23:37.313 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.STAT_OBJECT 2023-09-22T13:23:37.313 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.STAT_OBJECT (0 ms) 2023-09-22T13:23:37.313 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.WRITE_READ_VERIFY 2023-09-22T13:23:37.313 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.WRITE_READ_VERIFY (0 ms) 2023-09-22T13:23:37.313 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.WRITEV 2023-09-22T13:23:37.313 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.WRITEV (0 ms) 2023-09-22T13:23:37.314 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.READV 2023-09-22T13:23:37.314 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.READV (0 ms) 2023-09-22T13:23:37.314 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.READV_AFTER_WRITEV 2023-09-22T13:23:37.314 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.READV_AFTER_WRITEV (0 ms) 2023-09-22T13:23:37.314 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.DELETE_OBJECT 2023-09-22T13:23:37.315 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.DELETE_OBJECT (4 ms) 2023-09-22T13:23:37.315 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-09-22T13:23:37.344 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.DELETE_BUCKET (29 ms) 2023-09-22T13:23:37.345 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.CLEANUP 2023-09-22T13:23:37.345 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-09-22T13:23:37.345 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.UMOUNT 2023-09-22T13:23:37.345 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-09-22T13:23:37.345 INFO:tasks.workunit.client.0.smithi096.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-09-22T13:23:37.345 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:37.344+0000 7fa8fa524540 -1 shutting down 2023-09-22T13:23:37.345 INFO:tasks.workunit.client.0.smithi096.stderr:2023-09-22T13:23:37.344+0000 7fa811ffb700 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 149247, si_uid : 1000, si_addr0x3e8000246ff, si_status0 2023-09-22T13:23:37.405 INFO:tasks.workunit.client.0.smithi096.stdout:[ OK ] LibRGW.SHUTDOWN (61 ms) 2023-09-22T13:23:37.406 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] 19 tests from LibRGW (194 ms total) 2023-09-22T13:23:37.406 INFO:tasks.workunit.client.0.smithi096.stdout: 2023-09-22T13:23:37.406 INFO:tasks.workunit.client.0.smithi096.stdout:[----------] Global test environment tear-down 2023-09-22T13:23:37.406 INFO:tasks.workunit.client.0.smithi096.stdout:[==========] 19 tests from 1 test suite ran. (194 ms total) 2023-09-22T13:23:37.406 INFO:tasks.workunit.client.0.smithi096.stdout:[ PASSED ] 19 tests. 2023-09-22T13:23:37.408 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-09-22T13:23:37.408 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-09-22T13:23:37.436 INFO:tasks.workunit:Stopping ['cls/test_cls_lock.sh', 'cls/test_cls_log.sh', 'cls/test_cls_refcount.sh', 'cls/test_cls_rgw.sh', 'cls/test_cls_cmpomap.sh', 'cls/test_cls_2pc_queue.sh', 'rgw/test_rgw_gc_log.sh', 'rgw/test_rgw_obj.sh', 'rgw/test_rgw_throttle.sh', 'rgw/test_librgw_file.sh'] on client.0... 2023-09-22T13:23:37.437 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-09-22T13:23:37.735 DEBUG:teuthology.parallel:result is None 2023-09-22T13:23:37.735 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-09-22T13:23:37.763 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-09-22T13:23:37.763 DEBUG:teuthology.orchestra.run.smithi096:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-09-22T13:23:37.821 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-09-22T13:23:37.821 INFO:teuthology.run_tasks:Running task ragweed... 2023-09-22T13:23:37.836 DEBUG:tasks.ragweed:ragweed config is {'client.0': {'default-branch': 'ceph-pacific', 'rgw_server': 'client.0', 'stages': 'prepare,check'}} 2023-09-22T13:23:37.836 INFO:tasks.ragweed:Downloading ragweed... 2023-09-22T13:23:37.837 INFO:tasks.ragweed:Using branch 'wip-cfsnyder-bucket-check-pacific' for ragweed 2023-09-22T13:23:37.837 DEBUG:teuthology.orchestra.run.smithi096:> git clone -b wip-cfsnyder-bucket-check-pacific https://git.ceph.com/ragweed.git /home/ubuntu/cephtest/ragweed 2023-09-22T13:23:37.879 INFO:teuthology.orchestra.run.smithi096.stderr:Cloning into '/home/ubuntu/cephtest/ragweed'... 2023-09-22T13:23:37.958 INFO:teuthology.orchestra.run.smithi096.stderr:fatal: Remote branch wip-cfsnyder-bucket-check-pacific not found in upstream origin 2023-09-22T13:23:37.959 DEBUG:teuthology.orchestra.run:got remote process result: 128 2023-09-22T13:23:37.960 INFO:tasks.ragweed:Using branch 'ceph-pacific' for ragweed 2023-09-22T13:23:37.960 DEBUG:teuthology.orchestra.run.smithi096:> git clone -b ceph-pacific https://git.ceph.com/ragweed.git /home/ubuntu/cephtest/ragweed 2023-09-22T13:23:38.018 INFO:teuthology.orchestra.run.smithi096.stderr:Cloning into '/home/ubuntu/cephtest/ragweed'... 2023-09-22T13:23:38.259 INFO:tasks.ragweed:Creating rgw users... 2023-09-22T13:23:38.260 DEBUG:tasks.ragweed:Creating user ragweed.client.0 on client.0 2023-09-22T13:23:38.260 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid ragweed.client.0 --display-name 'Mr. ragweed.client.0' --access-key UHBQXDOOOUERCQBLIDRW --secret OjeoZyEFterJs1A2Pvkl8zzjO7XXsxZDUg8DB9FjGAT8pT+s2OMWVQ== --email ragweed.client.0+test@test.test --system false 2023-09-22T13:23:38.348 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:23:38.348 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:23:38.354 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.354+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.355 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.355+0000 7f82aed5f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.355 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.355+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.356 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.356+0000 7f82aed5f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.356 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.356+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.356 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.356+0000 7f82aed5f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.356 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.356+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.357 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.357+0000 7f82aed5f500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:23:38.569 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.568+0000 7f82aed5f500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:23:38.569 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.568+0000 7f82aed5f500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:23:38.569 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.568+0000 7f82aed5f500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:23:38.569 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.568+0000 7f82aed5f500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:23:38.569 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.568+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.570 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.569+0000 7f82aed5f500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:23:38.570 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.569+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.569+0000 7f82aed5f500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:23:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.569+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.569+0000 7f82aed5f500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:23:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.569+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.570+0000 7f82aed5f500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:23:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.570+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.570+0000 7f82aed5f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.570+0000 7f82aed5f500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:23:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.570+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.571 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.570+0000 7f82aed5f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.572 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.570+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.573 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.571+0000 7f82aed5f500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:23:38.573 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.571+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.573 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.571+0000 7f82aed5f500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:23:38.573 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.571+0000 7f82aed5f500 20 zonegroup default 2023-09-22T13:23:38.573 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.571+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.571+0000 7f82aed5f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.571+0000 7f82aed5f500 10 Cannot find current period zone using local zone 2023-09-22T13:23:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.571+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.572+0000 7f82aed5f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.572+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.572+0000 7f82aed5f500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:23:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.572+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.572+0000 7f82aed5f500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:23:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.572+0000 7f82aed5f500 20 zone default found 2023-09-22T13:23:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.572+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.574 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.573+0000 7f82aed5f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.575 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.573+0000 7f82aed5f500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.575 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.573+0000 7f82aed5f500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.575 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.573+0000 7f82aed5f500 20 started sync module instance, tier type = 2023-09-22T13:23:38.575 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.573+0000 7f82aed5f500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:23:38.579 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.579+0000 7f829a241700 0 -- 172.21.15.96:0/483511809 >> [v2:172.21.15.96:6824/140822,v1:172.21.15.96:6825/140822] conn(0x5643305287b0 msgr2=0x56433052aba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:23:38.581 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.581+0000 7f82aed5f500 20 add_watcher() i=0 2023-09-22T13:23:38.581 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.581+0000 7f82aed5f500 20 add_watcher() i=1 2023-09-22T13:23:38.581 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.581+0000 7f82aed5f500 20 add_watcher() i=2 2023-09-22T13:23:38.581 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.581+0000 7f82aed5f500 20 add_watcher() i=3 2023-09-22T13:23:38.581 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.581+0000 7f82aed5f500 20 add_watcher() i=4 2023-09-22T13:23:38.581 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.581+0000 7f82aed5f500 20 add_watcher() i=5 2023-09-22T13:23:38.582 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.581+0000 7f82aed5f500 20 add_watcher() i=6 2023-09-22T13:23:38.582 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.581+0000 7f82aed5f500 20 add_watcher() i=7 2023-09-22T13:23:38.582 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.581+0000 7f82aed5f500 2 all 8 watchers are set, enabling cache 2023-09-22T13:23:38.583 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.582+0000 7f826bfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:23:38.583 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.583+0000 7f82aed5f500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:23:38.583 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.583+0000 7f82aed5f500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:23:38.585 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.585+0000 7f829aa42700 0 -- 172.21.15.96:0/2072123973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x564330544f00 msgr2=0x56433054b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:23:38.588 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.587+0000 7f82aed5f500 5 note: GC not initialized 2023-09-22T13:23:38.589 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.588+0000 7f81daffd700 20 reqs_thread_entry: start 2023-09-22T13:23:38.589 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.588+0000 7f81d8ff9700 20 reqs_thread_entry: start 2023-09-22T13:23:38.589 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.588+0000 7f82aed5f500 20 init_complete bucket index max shards: 11 2023-09-22T13:23:38.589 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.588+0000 7f82aed5f500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:23:38.589 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.588+0000 7f81ceffd700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:24:08 2023 2023-09-22T13:23:38.589 INFO:teuthology.orchestra.run.smithi096.stderr: 2023-09-22T13:23:38.589 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.589+0000 7f81ce7fc700 20 reqs_thread_entry: start 2023-09-22T13:23:38.590 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.589+0000 7f82aed5f500 20 get_system_obj_state: rctx=0x7ffd763185a8 obj=default.rgw.meta:users.uid:ragweed.client.0 state=0x5643303e8810 s->prefetch_data=0 2023-09-22T13:23:38.590 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.589+0000 7f82aed5f500 10 cache get: name=default.rgw.meta+users.uid+ragweed.client.0 : miss 2023-09-22T13:23:38.592 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.589+0000 7f82aed5f500 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x0 2023-09-22T13:23:38.592 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.589+0000 7f82aed5f500 10 adding default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2023-09-22T13:23:38.592 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.589+0000 7f82aed5f500 20 get_system_obj_state: rctx=0x7ffd763185b8 obj=default.rgw.meta:users.email:ragweed.client.0+test@test.test state=0x5643303e8810 s->prefetch_data=0 2023-09-22T13:23:38.593 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.589+0000 7f82aed5f500 10 cache get: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test : miss 2023-09-22T13:23:38.593 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.590+0000 7f82aed5f500 10 cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x0 2023-09-22T13:23:38.593 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.590+0000 7f82aed5f500 10 adding default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2023-09-22T13:23:38.593 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.590+0000 7f82aed5f500 20 get_system_obj_state: rctx=0x7ffd763185b8 obj=default.rgw.meta:users.keys:UHBQXDOOOUERCQBLIDRW state=0x5643303e8810 s->prefetch_data=0 2023-09-22T13:23:38.593 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.590+0000 7f82aed5f500 10 cache get: name=default.rgw.meta+users.keys+UHBQXDOOOUERCQBLIDRW : miss 2023-09-22T13:23:38.593 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.590+0000 7f82aed5f500 10 cache put: name=default.rgw.meta+users.keys+UHBQXDOOOUERCQBLIDRW info.flags=0x0 2023-09-22T13:23:38.593 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.590+0000 7f82aed5f500 10 adding default.rgw.meta+users.keys+UHBQXDOOOUERCQBLIDRW to cache LRU end 2023-09-22T13:23:38.594 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.590+0000 7f82aed5f500 20 get_system_obj_state: rctx=0x7ffd76317eb8 obj=default.rgw.meta:users.keys:UHBQXDOOOUERCQBLIDRW state=0x5643303e8810 s->prefetch_data=0 2023-09-22T13:23:38.595 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.590+0000 7f82aed5f500 10 cache get: name=default.rgw.meta+users.keys+UHBQXDOOOUERCQBLIDRW : hit (negative entry) 2023-09-22T13:23:38.595 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.590+0000 7f82aed5f500 20 get_system_obj_state: rctx=0x7ffd763181f8 obj=default.rgw.meta:users.keys:UHBQXDOOOUERCQBLIDRW state=0x5643303e8810 s->prefetch_data=0 2023-09-22T13:23:38.595 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.590+0000 7f82aed5f500 10 cache get: name=default.rgw.meta+users.keys+UHBQXDOOOUERCQBLIDRW : hit (negative entry) 2023-09-22T13:23:38.595 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.591+0000 7f82aed5f500 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x17 2023-09-22T13:23:38.595 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.591+0000 7f82aed5f500 10 moving default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2023-09-22T13:23:38.595 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.591+0000 7f82aed5f500 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:ragweed.client.0, ofs0, ns] 2023-09-22T13:23:38.595 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.592+0000 7f828bfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 992137445394 cookie 94846573573264 notifier 5237 bl.length()=641 2023-09-22T13:23:38.596 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.592+0000 7f828bfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x17 2023-09-22T13:23:38.596 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.592+0000 7f828bfff700 10 rgw watcher librados: moving default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2023-09-22T13:23:38.597 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.593+0000 7f82aed5f500 10 cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x7 2023-09-22T13:23:38.598 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.593+0000 7f82aed5f500 10 moving default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2023-09-22T13:23:38.598 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.593+0000 7f82aed5f500 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.email:ragweed.client.0+test@test.test, ofs0, ns] 2023-09-22T13:23:38.598 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.593+0000 7f828bfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 992137445384 cookie 94846573568048 notifier 5237 bl.length()=202 2023-09-22T13:23:38.598 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.593+0000 7f828bfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x7 2023-09-22T13:23:38.598 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.593+0000 7f828bfff700 10 rgw watcher librados: moving default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2023-09-22T13:23:38.598 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.595+0000 7f82aed5f500 10 cache put: name=default.rgw.meta+users.keys+UHBQXDOOOUERCQBLIDRW info.flags=0x7 2023-09-22T13:23:38.598 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.595+0000 7f82aed5f500 10 moving default.rgw.meta+users.keys+UHBQXDOOOUERCQBLIDRW to cache LRU end 2023-09-22T13:23:38.598 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.595+0000 7f82aed5f500 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 0, obj: default.rgw.meta:users.keys:UHBQXDOOOUERCQBLIDRW, ofs0, ns] 2023-09-22T13:23:38.599 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.596+0000 7f828bfff700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 992137445387 cookie 94846573579920 notifier 5237 bl.length()=190 2023-09-22T13:23:38.599 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.596+0000 7f828bfff700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+UHBQXDOOOUERCQBLIDRW info.flags=0x7 2023-09-22T13:23:38.599 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.596+0000 7f828bfff700 10 rgw watcher librados: moving default.rgw.meta+users.keys+UHBQXDOOOUERCQBLIDRW to cache LRU end 2023-09-22T13:23:38.599 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2023-09-22T13:23:38.599 INFO:teuthology.orchestra.run.smithi096.stdout: "user_id": "ragweed.client.0", 2023-09-22T13:23:38.599 INFO:teuthology.orchestra.run.smithi096.stdout: "display_name": "Mr. ragweed.client.0", 2023-09-22T13:23:38.599 INFO:teuthology.orchestra.run.smithi096.stdout: "email": "ragweed.client.0+test@test.test", 2023-09-22T13:23:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: "suspended": 0, 2023-09-22T13:23:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: "max_buckets": 1000, 2023-09-22T13:23:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: "subusers": [], 2023-09-22T13:23:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: "keys": [ 2023-09-22T13:23:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2023-09-22T13:23:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: "user": "ragweed.client.0", 2023-09-22T13:23:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: "access_key": "UHBQXDOOOUERCQBLIDRW", 2023-09-22T13:23:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: "secret_key": "OjeoZyEFterJs1A2Pvkl8zzjO7XXsxZDUg8DB9FjGAT8pT+s2OMWVQ==" 2023-09-22T13:23:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: } 2023-09-22T13:23:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2023-09-22T13:23:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: "swift_keys": [], 2023-09-22T13:23:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": [], 2023-09-22T13:23:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: "op_mask": "read, write, delete", 2023-09-22T13:23:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: "default_placement": "", 2023-09-22T13:23:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: "default_storage_class": "", 2023-09-22T13:23:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: "placement_tags": [], 2023-09-22T13:23:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: "bucket_quota": { 2023-09-22T13:23:38.603 INFO:teuthology.orchestra.run.smithi096.stdout: "enabled": false, 2023-09-22T13:23:38.603 INFO:teuthology.orchestra.run.smithi096.stdout: "check_on_raw": false, 2023-09-22T13:23:38.603 INFO:teuthology.orchestra.run.smithi096.stdout: "max_size": -1, 2023-09-22T13:23:38.603 INFO:teuthology.orchestra.run.smithi096.stdout: "max_size_kb": 0, 2023-09-22T13:23:38.603 INFO:teuthology.orchestra.run.smithi096.stdout: "max_objects": -1 2023-09-22T13:23:38.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2023-09-22T13:23:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: "user_quota": { 2023-09-22T13:23:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: "enabled": false, 2023-09-22T13:23:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: "check_on_raw": false, 2023-09-22T13:23:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: "max_size": -1, 2023-09-22T13:23:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: "max_size_kb": 0, 2023-09-22T13:23:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: "max_objects": -1 2023-09-22T13:23:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2023-09-22T13:23:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: "temp_url_keys": [], 2023-09-22T13:23:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "rgw", 2023-09-22T13:23:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: "mfa_ids": [] 2023-09-22T13:23:38.605 INFO:teuthology.orchestra.run.smithi096.stdout:} 2023-09-22T13:23:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:23:38.607 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.597+0000 7f82aed5f500 20 remove_watcher() i=0 2023-09-22T13:23:38.607 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.597+0000 7f82aed5f500 2 removed watcher, disabling cache 2023-09-22T13:23:38.607 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.598+0000 7f82aed5f500 20 remove_watcher() i=1 2023-09-22T13:23:38.607 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.599+0000 7f82aed5f500 20 remove_watcher() i=2 2023-09-22T13:23:38.607 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.600+0000 7f82aed5f500 20 remove_watcher() i=3 2023-09-22T13:23:38.607 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.600+0000 7f82aed5f500 20 remove_watcher() i=4 2023-09-22T13:23:38.608 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.601+0000 7f82aed5f500 20 remove_watcher() i=5 2023-09-22T13:23:38.608 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.602+0000 7f82aed5f500 20 remove_watcher() i=6 2023-09-22T13:23:38.608 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.603+0000 7f82aed5f500 20 remove_watcher() i=7 2023-09-22T13:23:38.608 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.603+0000 7f826bfdf700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:23:38.609 DEBUG:tasks.ragweed:Creating user sysuser.client.0 on client.0 2023-09-22T13:23:38.610 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid sysuser.client.0 --display-name 'Mr. sysuser.client.0' --access-key DSXNSOVRUDCFWLDQHZKC --secret qHdeEmz4nZmu9eBC1PRpmTqjhh6VVUdFaZNSws9DsJT4XZFC9aEf5Q== --email sysuser.client.0+test@test.test --system true 2023-09-22T13:23:38.659 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setuser ceph since I am not root 2023-09-22T13:23:38.659 INFO:teuthology.orchestra.run.smithi096.stderr:ignoring --setgroup ceph since I am not root 2023-09-22T13:23:38.665 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.665+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.666 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.666+0000 7f3d481f9500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.666 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.666+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.667 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.667+0000 7f3d481f9500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.667 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.667+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.667 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.667+0000 7f3d481f9500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.668 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.667+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.668 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.668+0000 7f3d481f9500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:23:38.679 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.678+0000 7f3d481f9500 20 RGWRados::pool_iterate: got zonegroup_info.570f96c8-d368-4dce-92ba-9b1ab2b076d7 2023-09-22T13:23:38.679 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.678+0000 7f3d481f9500 20 RGWRados::pool_iterate: got zone_info.42123edd-4a97-43ec-80bc-fc5350f88c11 2023-09-22T13:23:38.679 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.678+0000 7f3d481f9500 20 RGWRados::pool_iterate: got zone_names.default 2023-09-22T13:23:38.679 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.678+0000 7f3d481f9500 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-22T13:23:38.680 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.678+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.680 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.678+0000 7f3d481f9500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:23:38.680 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.678+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.680 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.679+0000 7f3d481f9500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:23:38.680 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.679+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.680 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.679+0000 7f3d481f9500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:23:38.680 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.679+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.682 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.679+0000 7f3d481f9500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:23:38.682 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.679+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.682 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.679+0000 7f3d481f9500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.682 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.679+0000 7f3d481f9500 10 cannot find current period zonegroup using local zonegroup 2023-09-22T13:23:38.682 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.679+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.682 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.680+0000 7f3d481f9500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.682 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.680+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.683 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.680+0000 7f3d481f9500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:23:38.683 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.680+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.683 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.680+0000 7f3d481f9500 20 rados_obj.operate() r=0 bl.length=392 2023-09-22T13:23:38.683 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.680+0000 7f3d481f9500 20 zonegroup default 2023-09-22T13:23:38.683 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.680+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.683 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.680+0000 7f3d481f9500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.683 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.680+0000 7f3d481f9500 10 Cannot find current period zone using local zone 2023-09-22T13:23:38.683 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.680+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.683 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.681+0000 7f3d481f9500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.683 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.681+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.683 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.681+0000 7f3d481f9500 20 rados_obj.operate() r=0 bl.length=46 2023-09-22T13:23:38.683 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.681+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.684 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.681+0000 7f3d481f9500 20 rados_obj.operate() r=0 bl.length=1019 2023-09-22T13:23:38.684 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.681+0000 7f3d481f9500 20 zone default found 2023-09-22T13:23:38.684 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.681+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.684 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.681+0000 7f3d481f9500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.684 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.681+0000 7f3d481f9500 20 rados->read ofs=0 len=0 2023-09-22T13:23:38.684 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.682+0000 7f3d481f9500 20 rados_obj.operate() r=-2 bl.length=0 2023-09-22T13:23:38.684 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.682+0000 7f3d481f9500 20 started sync module instance, tier type = 2023-09-22T13:23:38.684 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.682+0000 7f3d481f9500 20 started zone id=42123edd-4a97-43ec-80bc-fc5350f88c11 (name=default) with tier type = 2023-09-22T13:23:38.689 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.689+0000 7f3d481f9500 20 add_watcher() i=0 2023-09-22T13:23:38.689 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.689+0000 7f3d481f9500 20 add_watcher() i=1 2023-09-22T13:23:38.689 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.689+0000 7f3d481f9500 20 add_watcher() i=2 2023-09-22T13:23:38.689 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.689+0000 7f3d481f9500 20 add_watcher() i=3 2023-09-22T13:23:38.689 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.689+0000 7f3d481f9500 20 add_watcher() i=4 2023-09-22T13:23:38.689 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.689+0000 7f3d481f9500 20 add_watcher() i=5 2023-09-22T13:23:38.689 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.689+0000 7f3d481f9500 20 add_watcher() i=6 2023-09-22T13:23:38.690 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.689+0000 7f3d481f9500 20 add_watcher() i=7 2023-09-22T13:23:38.690 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.689+0000 7f3d481f9500 2 all 8 watchers are set, enabling cache 2023-09-22T13:23:38.690 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.690+0000 7f3d32eda700 0 -- 172.21.15.96:0/519976787 >> [v2:172.21.15.96:6816/140810,v1:172.21.15.96:6817/140810] conn(0x55ec83a8a880 msgr2=0x55ec83a8cc70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:23:38.691 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.691+0000 7f3c7ffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:23:38.692 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.691+0000 7f3d481f9500 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-22T13:23:38.692 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.691+0000 7f3d481f9500 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-22T13:23:38.695 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.695+0000 7f3d32eda700 0 -- 172.21.15.96:0/1617262159 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55ec83aab670 msgr2=0x55ec83ab8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-09-22T13:23:38.697 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.697+0000 7f3d481f9500 5 note: GC not initialized 2023-09-22T13:23:38.699 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.698+0000 7f3c6effd700 20 reqs_thread_entry: start 2023-09-22T13:23:38.701 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.698+0000 7f3c6cff9700 20 reqs_thread_entry: start 2023-09-22T13:23:38.701 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.698+0000 7f3d481f9500 20 init_complete bucket index max shards: 11 2023-09-22T13:23:38.701 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.698+0000 7f3d481f9500 10 rgw notify: Started notification manager with: 1 workers 2023-09-22T13:23:38.701 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.698+0000 7f3c5effd700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 22 13:24:08 2023 2023-09-22T13:23:38.701 INFO:teuthology.orchestra.run.smithi096.stderr: 2023-09-22T13:23:38.701 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.698+0000 7f3c5e7fc700 20 reqs_thread_entry: start 2023-09-22T13:23:38.701 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.698+0000 7f3d481f9500 20 get_system_obj_state: rctx=0x7ffc835bbb58 obj=default.rgw.meta:users.uid:sysuser.client.0 state=0x55ec8394c810 s->prefetch_data=0 2023-09-22T13:23:38.701 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.698+0000 7f3d481f9500 10 cache get: name=default.rgw.meta+users.uid+sysuser.client.0 : miss 2023-09-22T13:23:38.703 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.699+0000 7f3d481f9500 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x0 2023-09-22T13:23:38.703 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.699+0000 7f3d481f9500 10 adding default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2023-09-22T13:23:38.703 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.699+0000 7f3d481f9500 20 get_system_obj_state: rctx=0x7ffc835bbb68 obj=default.rgw.meta:users.email:sysuser.client.0+test@test.test state=0x55ec8394c810 s->prefetch_data=0 2023-09-22T13:23:38.703 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.699+0000 7f3d481f9500 10 cache get: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test : miss 2023-09-22T13:23:38.703 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.699+0000 7f3d481f9500 10 cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x0 2023-09-22T13:23:38.703 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.699+0000 7f3d481f9500 10 adding default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2023-09-22T13:23:38.703 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.699+0000 7f3d481f9500 20 get_system_obj_state: rctx=0x7ffc835bbb68 obj=default.rgw.meta:users.keys:DSXNSOVRUDCFWLDQHZKC state=0x55ec8394c810 s->prefetch_data=0 2023-09-22T13:23:38.703 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.699+0000 7f3d481f9500 10 cache get: name=default.rgw.meta+users.keys+DSXNSOVRUDCFWLDQHZKC : miss 2023-09-22T13:23:38.704 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.700+0000 7f3d481f9500 10 cache put: name=default.rgw.meta+users.keys+DSXNSOVRUDCFWLDQHZKC info.flags=0x0 2023-09-22T13:23:38.704 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.700+0000 7f3d481f9500 10 adding default.rgw.meta+users.keys+DSXNSOVRUDCFWLDQHZKC to cache LRU end 2023-09-22T13:23:38.704 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.700+0000 7f3d481f9500 20 get_system_obj_state: rctx=0x7ffc835bb468 obj=default.rgw.meta:users.keys:DSXNSOVRUDCFWLDQHZKC state=0x55ec8394c810 s->prefetch_data=0 2023-09-22T13:23:38.704 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.700+0000 7f3d481f9500 10 cache get: name=default.rgw.meta+users.keys+DSXNSOVRUDCFWLDQHZKC : hit (negative entry) 2023-09-22T13:23:38.704 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.700+0000 7f3d481f9500 20 get_system_obj_state: rctx=0x7ffc835bb7a8 obj=default.rgw.meta:users.keys:DSXNSOVRUDCFWLDQHZKC state=0x55ec8394c810 s->prefetch_data=0 2023-09-22T13:23:38.704 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.700+0000 7f3d481f9500 10 cache get: name=default.rgw.meta+users.keys+DSXNSOVRUDCFWLDQHZKC : hit (negative entry) 2023-09-22T13:23:38.704 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.701+0000 7f3d481f9500 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x17 2023-09-22T13:23:38.704 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.701+0000 7f3d481f9500 10 moving default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2023-09-22T13:23:38.704 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.701+0000 7f3d481f9500 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.uid:sysuser.client.0, ofs0, ns] 2023-09-22T13:23:38.704 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.701+0000 7f3d316d7700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 992137445385 cookie 94474309576528 notifier 5511 bl.length()=641 2023-09-22T13:23:38.704 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.701+0000 7f3d316d7700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x17 2023-09-22T13:23:38.704 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.701+0000 7f3d316d7700 10 rgw watcher librados: moving default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2023-09-22T13:23:38.704 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.702+0000 7f3d481f9500 10 cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x7 2023-09-22T13:23:38.705 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.702+0000 7f3d481f9500 10 moving default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2023-09-22T13:23:38.705 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.702+0000 7f3d481f9500 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.email:sysuser.client.0+test@test.test, ofs0, ns] 2023-09-22T13:23:38.705 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.703+0000 7f3d316d7700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 992137445395 cookie 94474309582784 notifier 5511 bl.length()=202 2023-09-22T13:23:38.706 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.703+0000 7f3d316d7700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x7 2023-09-22T13:23:38.706 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.703+0000 7f3d316d7700 10 rgw watcher librados: moving default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2023-09-22T13:23:38.706 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.704+0000 7f3d481f9500 10 cache put: name=default.rgw.meta+users.keys+DSXNSOVRUDCFWLDQHZKC info.flags=0x7 2023-09-22T13:23:38.706 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.704+0000 7f3d481f9500 10 moving default.rgw.meta+users.keys+DSXNSOVRUDCFWLDQHZKC to cache LRU end 2023-09-22T13:23:38.706 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.704+0000 7f3d481f9500 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.keys:DSXNSOVRUDCFWLDQHZKC, ofs0, ns] 2023-09-22T13:23:38.706 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.704+0000 7f3d316d7700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 992137445466 cookie 94474309587200 notifier 5511 bl.length()=190 2023-09-22T13:23:38.706 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.704+0000 7f3d316d7700 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+DSXNSOVRUDCFWLDQHZKC info.flags=0x7 2023-09-22T13:23:38.706 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.704+0000 7f3d316d7700 10 rgw watcher librados: moving default.rgw.meta+users.keys+DSXNSOVRUDCFWLDQHZKC to cache LRU end 2023-09-22T13:23:38.706 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2023-09-22T13:23:38.707 INFO:teuthology.orchestra.run.smithi096.stdout: "user_id": "sysuser.client.0", 2023-09-22T13:23:38.707 INFO:teuthology.orchestra.run.smithi096.stdout: "display_name": "Mr. sysuser.client.0", 2023-09-22T13:23:38.707 INFO:teuthology.orchestra.run.smithi096.stdout: "email": "sysuser.client.0+test@test.test", 2023-09-22T13:23:38.707 INFO:teuthology.orchestra.run.smithi096.stdout: "suspended": 0, 2023-09-22T13:23:38.707 INFO:teuthology.orchestra.run.smithi096.stdout: "max_buckets": 1000, 2023-09-22T13:23:38.707 INFO:teuthology.orchestra.run.smithi096.stdout: "subusers": [], 2023-09-22T13:23:38.707 INFO:teuthology.orchestra.run.smithi096.stdout: "keys": [ 2023-09-22T13:23:38.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2023-09-22T13:23:38.707 INFO:teuthology.orchestra.run.smithi096.stdout: "user": "sysuser.client.0", 2023-09-22T13:23:38.707 INFO:teuthology.orchestra.run.smithi096.stdout: "access_key": "DSXNSOVRUDCFWLDQHZKC", 2023-09-22T13:23:38.707 INFO:teuthology.orchestra.run.smithi096.stdout: "secret_key": "qHdeEmz4nZmu9eBC1PRpmTqjhh6VVUdFaZNSws9DsJT4XZFC9aEf5Q==" 2023-09-22T13:23:38.707 INFO:teuthology.orchestra.run.smithi096.stdout: } 2023-09-22T13:23:38.708 INFO:teuthology.orchestra.run.smithi096.stdout: ], 2023-09-22T13:23:38.708 INFO:teuthology.orchestra.run.smithi096.stdout: "swift_keys": [], 2023-09-22T13:23:38.708 INFO:teuthology.orchestra.run.smithi096.stdout: "caps": [], 2023-09-22T13:23:38.708 INFO:teuthology.orchestra.run.smithi096.stdout: "op_mask": "read, write, delete", 2023-09-22T13:23:38.708 INFO:teuthology.orchestra.run.smithi096.stdout: "system": "true", 2023-09-22T13:23:38.708 INFO:teuthology.orchestra.run.smithi096.stdout: "default_placement": "", 2023-09-22T13:23:38.708 INFO:teuthology.orchestra.run.smithi096.stdout: "default_storage_class": "", 2023-09-22T13:23:38.708 INFO:teuthology.orchestra.run.smithi096.stdout: "placement_tags": [], 2023-09-22T13:23:38.708 INFO:teuthology.orchestra.run.smithi096.stdout: "bucket_quota": { 2023-09-22T13:23:38.708 INFO:teuthology.orchestra.run.smithi096.stdout: "enabled": false, 2023-09-22T13:23:38.708 INFO:teuthology.orchestra.run.smithi096.stdout: "check_on_raw": false, 2023-09-22T13:23:38.708 INFO:teuthology.orchestra.run.smithi096.stdout: "max_size": -1, 2023-09-22T13:23:38.708 INFO:teuthology.orchestra.run.smithi096.stdout: "max_size_kb": 0, 2023-09-22T13:23:38.709 INFO:teuthology.orchestra.run.smithi096.stdout: "max_objects": -1 2023-09-22T13:23:38.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2023-09-22T13:23:38.709 INFO:teuthology.orchestra.run.smithi096.stdout: "user_quota": { 2023-09-22T13:23:38.709 INFO:teuthology.orchestra.run.smithi096.stdout: "enabled": false, 2023-09-22T13:23:38.709 INFO:teuthology.orchestra.run.smithi096.stdout: "check_on_raw": false, 2023-09-22T13:23:38.709 INFO:teuthology.orchestra.run.smithi096.stdout: "max_size": -1, 2023-09-22T13:23:38.709 INFO:teuthology.orchestra.run.smithi096.stdout: "max_size_kb": 0, 2023-09-22T13:23:38.709 INFO:teuthology.orchestra.run.smithi096.stdout: "max_objects": -1 2023-09-22T13:23:38.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2023-09-22T13:23:38.709 INFO:teuthology.orchestra.run.smithi096.stdout: "temp_url_keys": [], 2023-09-22T13:23:38.709 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "rgw", 2023-09-22T13:23:38.709 INFO:teuthology.orchestra.run.smithi096.stdout: "mfa_ids": [] 2023-09-22T13:23:38.709 INFO:teuthology.orchestra.run.smithi096.stdout:} 2023-09-22T13:23:38.710 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:23:38.710 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.706+0000 7f3d481f9500 20 remove_watcher() i=0 2023-09-22T13:23:38.710 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.706+0000 7f3d481f9500 2 removed watcher, disabling cache 2023-09-22T13:23:38.710 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.707+0000 7f3d481f9500 20 remove_watcher() i=1 2023-09-22T13:23:38.711 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.708+0000 7f3d481f9500 20 remove_watcher() i=2 2023-09-22T13:23:38.711 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.709+0000 7f3d481f9500 20 remove_watcher() i=3 2023-09-22T13:23:38.711 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.709+0000 7f3d481f9500 20 remove_watcher() i=4 2023-09-22T13:23:38.711 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.710+0000 7f3d481f9500 20 remove_watcher() i=5 2023-09-22T13:23:38.711 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.711+0000 7f3d481f9500 20 remove_watcher() i=6 2023-09-22T13:23:38.712 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.712+0000 7f3d481f9500 20 remove_watcher() i=7 2023-09-22T13:23:38.712 INFO:teuthology.orchestra.run.smithi096.stderr:2023-09-22T13:23:38.712+0000 7f3c7ffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-22T13:23:38.717 INFO:tasks.ragweed:Configuring ragweed... 2023-09-22T13:23:38.717 DEBUG:teuthology.orchestra.run.smithi096:> cd /home/ubuntu/cephtest/ragweed && ./bootstrap 2023-09-22T13:23:38.735 INFO:teuthology.orchestra.run.smithi096.stderr:+ set -e 2023-09-22T13:23:38.735 INFO:teuthology.orchestra.run.smithi096.stderr:+ '[' -f /etc/debian_version ']' 2023-09-22T13:23:38.735 INFO:teuthology.orchestra.run.smithi096.stderr:+ '[' -f /etc/redhat-release ']' 2023-09-22T13:23:38.735 INFO:teuthology.orchestra.run.smithi096.stderr:+ for package in python3-pip python3-virtualenv python3-devel libevent-devel libxml2-devel libxslt-devel zlib-devel 2023-09-22T13:23:38.735 INFO:teuthology.orchestra.run.smithi096.stderr:++ rpm -qa python3-pip 2023-09-22T13:23:39.879 INFO:teuthology.orchestra.run.smithi096.stderr:+ '[' python3-pip-9.0.3-22.el8.noarch == '' ']' 2023-09-22T13:23:39.879 INFO:teuthology.orchestra.run.smithi096.stderr:+ for package in python3-pip python3-virtualenv python3-devel libevent-devel libxml2-devel libxslt-devel zlib-devel 2023-09-22T13:23:39.879 INFO:teuthology.orchestra.run.smithi096.stderr:++ rpm -qa python3-virtualenv 2023-09-22T13:23:41.012 INFO:teuthology.orchestra.run.smithi096.stderr:+ '[' python3-virtualenv-15.1.0-21.module_el8.5.0+895+a459eca8.noarch == '' ']' 2023-09-22T13:23:41.012 INFO:teuthology.orchestra.run.smithi096.stderr:+ for package in python3-pip python3-virtualenv python3-devel libevent-devel libxml2-devel libxslt-devel zlib-devel 2023-09-22T13:23:41.012 INFO:teuthology.orchestra.run.smithi096.stderr:++ rpm -qa python3-devel 2023-09-22T13:23:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:+ '[' '' == '' ']' 2023-09-22T13:23:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:+ missing=python3-devel 2023-09-22T13:23:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:+ for package in python3-pip python3-virtualenv python3-devel libevent-devel libxml2-devel libxslt-devel zlib-devel 2023-09-22T13:23:42.148 INFO:teuthology.orchestra.run.smithi096.stderr:++ rpm -qa libevent-devel 2023-09-22T13:23:43.276 INFO:teuthology.orchestra.run.smithi096.stderr:+ '[' '' == '' ']' 2023-09-22T13:23:43.276 INFO:teuthology.orchestra.run.smithi096.stderr:+ missing='python3-devel libevent-devel' 2023-09-22T13:23:43.276 INFO:teuthology.orchestra.run.smithi096.stderr:+ for package in python3-pip python3-virtualenv python3-devel libevent-devel libxml2-devel libxslt-devel zlib-devel 2023-09-22T13:23:43.276 INFO:teuthology.orchestra.run.smithi096.stderr:++ rpm -qa libxml2-devel 2023-09-22T13:23:44.434 INFO:teuthology.orchestra.run.smithi096.stderr:+ '[' '' == '' ']' 2023-09-22T13:23:44.434 INFO:teuthology.orchestra.run.smithi096.stderr:+ missing='python3-devel libevent-devel libxml2-devel' 2023-09-22T13:23:44.434 INFO:teuthology.orchestra.run.smithi096.stderr:+ for package in python3-pip python3-virtualenv python3-devel libevent-devel libxml2-devel libxslt-devel zlib-devel 2023-09-22T13:23:44.434 INFO:teuthology.orchestra.run.smithi096.stderr:++ rpm -qa libxslt-devel 2023-09-22T13:23:45.569 INFO:teuthology.orchestra.run.smithi096.stderr:+ '[' '' == '' ']' 2023-09-22T13:23:45.569 INFO:teuthology.orchestra.run.smithi096.stderr:+ missing='python3-devel libevent-devel libxml2-devel libxslt-devel' 2023-09-22T13:23:45.569 INFO:teuthology.orchestra.run.smithi096.stderr:+ for package in python3-pip python3-virtualenv python3-devel libevent-devel libxml2-devel libxslt-devel zlib-devel 2023-09-22T13:23:45.569 INFO:teuthology.orchestra.run.smithi096.stderr:++ rpm -qa zlib-devel 2023-09-22T13:23:46.706 INFO:teuthology.orchestra.run.smithi096.stderr:+ '[' '' == '' ']' 2023-09-22T13:23:46.706 INFO:teuthology.orchestra.run.smithi096.stderr:+ missing='python3-devel libevent-devel libxml2-devel libxslt-devel zlib-devel' 2023-09-22T13:23:46.706 INFO:teuthology.orchestra.run.smithi096.stderr:+ '[' -n 'python3-devel libevent-devel libxml2-devel libxslt-devel zlib-devel' ']' 2023-09-22T13:23:46.706 INFO:teuthology.orchestra.run.smithi096.stderr:+ echo './bootstrap: missing required RPM packages. Installing via sudo.' 2023-09-22T13:23:46.706 INFO:teuthology.orchestra.run.smithi096.stderr:./bootstrap: missing required RPM packages. Installing via sudo. 2023-09-22T13:23:46.706 INFO:teuthology.orchestra.run.smithi096.stderr:+ sudo yum -y install python3-devel libevent-devel libxml2-devel libxslt-devel zlib-devel 2023-09-22T13:23:47.548 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:33:30 ago on Fri 22 Sep 2023 12:50:17 PM UTC. 2023-09-22T13:23:48.947 INFO:teuthology.orchestra.run.smithi096.stdout:Package python36-devel-3.6.8-38.module_el8.5.0+895+a459eca8.x86_64 is already installed. 2023-09-22T13:23:49.137 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2023-09-22T13:23:49.139 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:23:49.139 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2023-09-22T13:23:49.139 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:23:49.139 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2023-09-22T13:23:49.139 INFO:teuthology.orchestra.run.smithi096.stdout: libevent-devel x86_64 2.1.8-5.el8 appstream 104 k 2023-09-22T13:23:49.139 INFO:teuthology.orchestra.run.smithi096.stdout: libxml2-devel x86_64 2.9.7-16.el8 appstream 1.0 M 2023-09-22T13:23:49.139 INFO:teuthology.orchestra.run.smithi096.stdout: libxslt-devel x86_64 1.1.32-6.el8 appstream 323 k 2023-09-22T13:23:49.139 INFO:teuthology.orchestra.run.smithi096.stdout: zlib-devel x86_64 1.2.11-25.el8 baseos 59 k 2023-09-22T13:23:49.139 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrading: 2023-09-22T13:23:49.140 INFO:teuthology.orchestra.run.smithi096.stdout: zlib x86_64 1.2.11-25.el8 baseos 103 k 2023-09-22T13:23:49.140 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2023-09-22T13:23:49.140 INFO:teuthology.orchestra.run.smithi096.stdout: cmake-filesystem x86_64 3.20.2-5.el8 appstream 45 k 2023-09-22T13:23:49.140 INFO:teuthology.orchestra.run.smithi096.stdout: libgcrypt-devel x86_64 1.8.5-7.el8 baseos 150 k 2023-09-22T13:23:49.140 INFO:teuthology.orchestra.run.smithi096.stdout: libgpg-error-devel x86_64 1.31-1.el8 baseos 64 k 2023-09-22T13:23:49.140 INFO:teuthology.orchestra.run.smithi096.stdout: xz-devel x86_64 5.2.4-4.el8 baseos 63 k 2023-09-22T13:23:49.140 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:23:49.140 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2023-09-22T13:23:49.140 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2023-09-22T13:23:49.140 INFO:teuthology.orchestra.run.smithi096.stdout:Install 8 Packages 2023-09-22T13:23:49.140 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrade 1 Package 2023-09-22T13:23:49.140 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:23:49.141 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 1.9 M 2023-09-22T13:23:49.141 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2023-09-22T13:23:51.700 INFO:teuthology.orchestra.run.smithi096.stdout:(1/9): cmake-filesystem-3.20.2-5.el8.x86_64.rpm 19 kB/s | 45 kB 00:02 2023-09-22T13:23:52.301 INFO:teuthology.orchestra.run.smithi096.stdout:(2/9): libevent-devel-2.1.8-5.el8.x86_64.rpm 36 kB/s | 104 kB 00:02 2023-09-22T13:23:53.367 INFO:teuthology.orchestra.run.smithi096.stdout:(3/9): libgcrypt-devel-1.8.5-7.el8.x86_64.rpm 140 kB/s | 150 kB 00:01 2023-09-22T13:23:55.201 INFO:teuthology.orchestra.run.smithi096.stdout:(4/9): libgpg-error-devel-1.31-1.el8.x86_64.rpm 35 kB/s | 64 kB 00:01 2023-09-22T13:23:56.759 INFO:teuthology.orchestra.run.smithi096.stdout:(5/9): xz-devel-5.2.4-4.el8.x86_64.rpm 40 kB/s | 63 kB 00:01 2023-09-22T13:24:04.242 INFO:teuthology.orchestra.run.smithi096.stdout:(6/9): libxslt-devel-1.1.32-6.el8.x86_64.rpm 26 kB/s | 323 kB 00:12 2023-09-22T13:24:04.975 INFO:teuthology.orchestra.run.smithi096.stdout:(7/9): zlib-1.2.11-25.el8.x86_64.rpm 141 kB/s | 103 kB 00:00 2023-09-22T13:24:29.338 INFO:teuthology.orchestra.run.smithi096.stdout:(8/9): zlib-devel-1.2.11-25.el8.x86_64.rpm 1.8 kB/s | 59 kB 00:32 2023-09-22T13:24:56.459 INFO:teuthology.orchestra.run.smithi096.stdout:(9/9): libxml2-devel-2.9.7-16.el8.x86_64.rpm 16 kB/s | 1.0 MB 01:07 2023-09-22T13:24:56.459 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2023-09-22T13:24:56.460 INFO:teuthology.orchestra.run.smithi096.stdout:Total 29 kB/s | 1.9 MB 01:07 2023-09-22T13:24:56.548 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2023-09-22T13:24:56.677 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2023-09-22T13:24:56.677 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2023-09-22T13:24:56.792 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2023-09-22T13:24:56.795 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2023-09-22T13:24:57.170 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2023-09-22T13:24:57.316 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libgpg-error-devel-1.31-1.el8.x86_64 1/10 2023-09-22T13:24:57.355 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libgpg-error-devel-1.31-1.el8.x86_64 1/10 2023-09-22T13:24:57.541 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libgcrypt-devel-1.8.5-7.el8.x86_64 2/10 2023-09-22T13:24:57.558 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libgcrypt-devel-1.8.5-7.el8.x86_64 2/10 2023-09-22T13:24:57.772 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : zlib-1.2.11-25.el8.x86_64 3/10 2023-09-22T13:24:57.971 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : zlib-devel-1.2.11-25.el8.x86_64 4/10 2023-09-22T13:24:58.172 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xz-devel-5.2.4-4.el8.x86_64 5/10 2023-09-22T13:24:58.616 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cmake-filesystem-3.20.2-5.el8.x86_64 6/10 2023-09-22T13:24:58.930 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libxml2-devel-2.9.7-16.el8.x86_64 7/10 2023-09-22T13:24:59.178 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libxslt-devel-1.1.32-6.el8.x86_64 8/10 2023-09-22T13:24:59.349 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libevent-devel-2.1.8-5.el8.x86_64 9/10 2023-09-22T13:24:59.506 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : zlib-1.2.11-21.el8.x86_64 10/10 2023-09-22T13:24:59.807 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: zlib-1.2.11-21.el8.x86_64 10/10 2023-09-22T13:24:59.807 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cmake-filesystem-3.20.2-5.el8.x86_64 1/10 2023-09-22T13:24:59.807 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libevent-devel-2.1.8-5.el8.x86_64 2/10 2023-09-22T13:24:59.807 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libxml2-devel-2.9.7-16.el8.x86_64 3/10 2023-09-22T13:24:59.807 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libxslt-devel-1.1.32-6.el8.x86_64 4/10 2023-09-22T13:24:59.807 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libgcrypt-devel-1.8.5-7.el8.x86_64 5/10 2023-09-22T13:24:59.807 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libgpg-error-devel-1.31-1.el8.x86_64 6/10 2023-09-22T13:24:59.807 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xz-devel-5.2.4-4.el8.x86_64 7/10 2023-09-22T13:24:59.807 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : zlib-devel-1.2.11-25.el8.x86_64 8/10 2023-09-22T13:24:59.807 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : zlib-1.2.11-25.el8.x86_64 9/10 2023-09-22T13:25:00.509 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : zlib-1.2.11-21.el8.x86_64 10/10 2023-09-22T13:25:00.509 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:25:00.509 INFO:teuthology.orchestra.run.smithi096.stdout:Upgraded: 2023-09-22T13:25:00.509 INFO:teuthology.orchestra.run.smithi096.stdout: zlib-1.2.11-25.el8.x86_64 2023-09-22T13:25:00.509 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2023-09-22T13:25:00.509 INFO:teuthology.orchestra.run.smithi096.stdout: cmake-filesystem-3.20.2-5.el8.x86_64 libevent-devel-2.1.8-5.el8.x86_64 2023-09-22T13:25:00.509 INFO:teuthology.orchestra.run.smithi096.stdout: libgcrypt-devel-1.8.5-7.el8.x86_64 libgpg-error-devel-1.31-1.el8.x86_64 2023-09-22T13:25:00.510 INFO:teuthology.orchestra.run.smithi096.stdout: libxml2-devel-2.9.7-16.el8.x86_64 libxslt-devel-1.1.32-6.el8.x86_64 2023-09-22T13:25:00.510 INFO:teuthology.orchestra.run.smithi096.stdout: xz-devel-5.2.4-4.el8.x86_64 zlib-devel-1.2.11-25.el8.x86_64 2023-09-22T13:25:00.510 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:25:00.510 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2023-09-22T13:25:00.751 INFO:teuthology.orchestra.run.smithi096.stderr:+ sudo pip3 install --upgrade --trusted-host apt-mirror.front.sepia.ceph.com setuptools cffi cachecontrol 2023-09-22T13:25:01.069 INFO:teuthology.orchestra.run.smithi096.stderr:WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead. 2023-09-22T13:25:01.845 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting setuptools 2023-09-22T13:25:02.324 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading https://files.pythonhosted.org/packages/b0/3a/88b210db68e56854d0bcf4b38e165e03be377e13907746f825790f3df5bf/setuptools-59.6.0-py3-none-any.whl (952kB) 2023-09-22T13:25:03.176 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting cffi 2023-09-22T13:25:03.430 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading https://files.pythonhosted.org/packages/3a/12/d6066828014b9ccb2bbb8e1d9dc28872d20669b65aeb4a86806a0757813f/cffi-1.15.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (402kB) 2023-09-22T13:25:03.594 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting cachecontrol 2023-09-22T13:25:03.804 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading https://files.pythonhosted.org/packages/72/a2/28e0ef082f7d78253aded97933e1d7b94bab3c5be366e8afd6513de4028e/CacheControl-0.12.14-py2.py3-none-any.whl 2023-09-22T13:25:04.049 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting pycparser (from cffi) 2023-09-22T13:25:04.101 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading https://files.pythonhosted.org/packages/62/d5/5f610ebe421e85889f2e55e33b7f9a6795bd982198517d912eb1c76e1a53/pycparser-2.21-py2.py3-none-any.whl (118kB) 2023-09-22T13:25:04.197 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting msgpack>=0.5.2 (from cachecontrol) 2023-09-22T13:25:04.538 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading https://files.pythonhosted.org/packages/dc/a1/eba11a0d4b764bc62966a565b470f8c6f38242723ba3057e9b5098678c30/msgpack-1.0.5.tar.gz (127kB) 2023-09-22T13:25:05.290 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting requests (from cachecontrol) 2023-09-22T13:25:05.391 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading https://files.pythonhosted.org/packages/2d/61/08076519c80041bc0ffa1a8af0cbd3bf3e2b62af10435d269a9d0f40564d/requests-2.27.1-py2.py3-none-any.whl (63kB) 2023-09-22T13:25:05.481 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting charset-normalizer~=2.0.0; python_version >= "3" (from requests->cachecontrol) 2023-09-22T13:25:05.845 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading https://files.pythonhosted.org/packages/06/b3/24afc8868eba069a7f03650ac750a778862dc34941a4bebeb58706715726/charset_normalizer-2.0.12-py3-none-any.whl 2023-09-22T13:25:06.148 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting certifi>=2017.4.17 (from requests->cachecontrol) 2023-09-22T13:25:06.211 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading https://files.pythonhosted.org/packages/4c/dd/2234eab22353ffc7d94e8d13177aaa050113286e93e7b40eae01fbf7c3d9/certifi-2023.7.22-py3-none-any.whl (158kB) 2023-09-22T13:25:06.453 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting idna<4,>=2.5; python_version >= "3" (from requests->cachecontrol) 2023-09-22T13:25:06.498 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl (61kB) 2023-09-22T13:25:06.709 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting urllib3<1.27,>=1.21.1 (from requests->cachecontrol) 2023-09-22T13:25:06.811 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading https://files.pythonhosted.org/packages/c5/05/c214b32d21c0b465506f95c4f28ccbcba15022e000b043b72b3df7728471/urllib3-1.26.16-py2.py3-none-any.whl (143kB) 2023-09-22T13:25:06.863 INFO:teuthology.orchestra.run.smithi096.stdout:Installing collected packages: setuptools, pycparser, cffi, msgpack, charset-normalizer, certifi, idna, urllib3, requests, cachecontrol 2023-09-22T13:25:07.228 INFO:teuthology.orchestra.run.smithi096.stdout: Running setup.py install for msgpack: started 2023-09-22T13:25:07.503 INFO:teuthology.orchestra.run.smithi096.stdout: Running setup.py install for msgpack: finished with status 'done' 2023-09-22T13:25:07.690 INFO:teuthology.orchestra.run.smithi096.stdout:Successfully installed cachecontrol-0.12.14 certifi-2023.7.22 cffi-1.15.1 charset-normalizer-2.0.12 idna-3.4 msgpack-1.0.5 pycparser-2.21 requests-2.27.1 setuptools-59.6.0 urllib3-1.26.16 2023-09-22T13:25:07.812 INFO:teuthology.orchestra.run.smithi096.stderr:+ virtualenv -p python3 --system-site-packages --download virtualenv 2023-09-22T13:25:08.177 INFO:teuthology.orchestra.run.smithi096.stdout:Using base prefix '/usr' 2023-09-22T13:25:08.177 INFO:teuthology.orchestra.run.smithi096.stdout:New python executable in /home/ubuntu/cephtest/ragweed/virtualenv/bin/python3 2023-09-22T13:25:08.177 INFO:teuthology.orchestra.run.smithi096.stdout:Also creating executable in /home/ubuntu/cephtest/ragweed/virtualenv/bin/python 2023-09-22T13:25:11.345 INFO:teuthology.orchestra.run.smithi096.stdout:Installing setuptools, pip, wheel...done. 2023-09-22T13:25:11.354 INFO:teuthology.orchestra.run.smithi096.stdout:Running virtualenv with interpreter /usr/bin/python3 2023-09-22T13:25:11.369 INFO:teuthology.orchestra.run.smithi096.stderr:+ ./virtualenv/bin/pip install --upgrade pip 2023-09-22T13:25:11.740 INFO:teuthology.orchestra.run.smithi096.stdout:Requirement already satisfied: pip in ./virtualenv/lib/python3.6/site-packages (21.3.1) 2023-09-22T13:25:12.353 INFO:teuthology.orchestra.run.smithi096.stderr:+ ./virtualenv/bin/pip install six 2023-09-22T13:25:12.719 INFO:teuthology.orchestra.run.smithi096.stdout:Requirement already satisfied: six in /usr/lib/python3.6/site-packages (1.11.0) 2023-09-22T13:25:12.902 INFO:teuthology.orchestra.run.smithi096.stderr:+ ./virtualenv/bin/pip install -I nose 2023-09-22T13:25:13.394 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting nose 2023-09-22T13:25:13.451 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading nose-1.3.7-py3-none-any.whl (154 kB) 2023-09-22T13:25:13.648 INFO:teuthology.orchestra.run.smithi096.stdout:Installing collected packages: nose 2023-09-22T13:25:13.752 INFO:teuthology.orchestra.run.smithi096.stdout:Successfully installed nose-1.3.7 2023-09-22T13:25:13.804 INFO:teuthology.orchestra.run.smithi096.stderr:+ ./virtualenv/bin/pip install setuptools 2023-09-22T13:25:14.175 INFO:teuthology.orchestra.run.smithi096.stdout:Requirement already satisfied: setuptools in ./virtualenv/lib/python3.6/site-packages (59.6.0) 2023-09-22T13:25:14.375 INFO:teuthology.orchestra.run.smithi096.stderr:+ ./virtualenv/bin/pip install -U -r requirements.txt 2023-09-22T13:25:14.744 INFO:teuthology.orchestra.run.smithi096.stdout:Requirement already satisfied: PyYAML in /usr/lib64/python3.6/site-packages (from -r requirements.txt (line 1)) (3.12) 2023-09-22T13:25:15.239 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting PyYAML 2023-09-22T13:25:15.297 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB) 2023-09-22T13:25:15.397 INFO:teuthology.orchestra.run.smithi096.stdout:Requirement already satisfied: nose>=1.0.0 in ./virtualenv/lib/python3.6/site-packages (from -r requirements.txt (line 2)) (1.3.7) 2023-09-22T13:25:15.521 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting boto!=2.46.0,>=2.6.0 2023-09-22T13:25:15.541 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading boto-2.49.0-py2.py3-none-any.whl (1.4 MB) 2023-09-22T13:25:15.693 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting munch>=1.0.0 2023-09-22T13:25:15.711 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading munch-4.0.0-py2.py3-none-any.whl (9.9 kB) 2023-09-22T13:25:16.160 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting gevent>=1.0 2023-09-22T13:25:16.194 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading gevent-22.10.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.8 MB) 2023-09-22T13:25:16.456 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting httplib2 2023-09-22T13:25:16.474 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading httplib2-0.22.0-py3-none-any.whl (96 kB) 2023-09-22T13:25:16.498 INFO:teuthology.orchestra.run.smithi096.stdout:Requirement already satisfied: lxml in /usr/lib64/python3.6/site-packages (from -r requirements.txt (line 7)) (4.2.3) 2023-09-22T13:25:17.028 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting lxml 2023-09-22T13:25:17.053 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading lxml-4.9.3-cp36-cp36m-manylinux_2_28_x86_64.whl (7.4 MB) 2023-09-22T13:25:17.471 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting importlib-metadata>=1.7.0 2023-09-22T13:25:17.488 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading importlib_metadata-4.8.3-py3-none-any.whl (17 kB) 2023-09-22T13:25:18.006 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting greenlet>=2.0.0 2023-09-22T13:25:18.027 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading greenlet-2.0.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (564 kB) 2023-09-22T13:25:18.424 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting zope.interface 2023-09-22T13:25:18.444 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading zope.interface-5.5.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (253 kB) 2023-09-22T13:25:18.523 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting zope.event 2023-09-22T13:25:18.540 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading zope.event-4.6-py2.py3-none-any.whl (6.8 kB) 2023-09-22T13:25:18.561 INFO:teuthology.orchestra.run.smithi096.stdout:Requirement already satisfied: setuptools in ./virtualenv/lib/python3.6/site-packages (from gevent>=1.0->-r requirements.txt (line 5)) (59.6.0) 2023-09-22T13:25:18.684 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 2023-09-22T13:25:18.701 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB) 2023-09-22T13:25:18.876 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting zipp>=0.5 2023-09-22T13:25:18.895 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading zipp-3.6.0-py3-none-any.whl (5.3 kB) 2023-09-22T13:25:18.990 INFO:teuthology.orchestra.run.smithi096.stdout:Collecting typing-extensions>=3.6.4 2023-09-22T13:25:19.017 INFO:teuthology.orchestra.run.smithi096.stdout: Downloading typing_extensions-4.1.1-py3-none-any.whl (26 kB) 2023-09-22T13:25:19.307 INFO:teuthology.orchestra.run.smithi096.stdout:Installing collected packages: zipp, typing-extensions, zope.interface, zope.event, pyparsing, importlib-metadata, greenlet, PyYAML, munch, lxml, httplib2, gevent, boto 2023-09-22T13:25:19.624 INFO:teuthology.orchestra.run.smithi096.stdout: Attempting uninstall: pyparsing 2023-09-22T13:25:19.625 INFO:teuthology.orchestra.run.smithi096.stdout: Found existing installation: pyparsing 2.1.10 2023-09-22T13:25:19.625 INFO:teuthology.orchestra.run.smithi096.stdout: Not uninstalling pyparsing at /usr/lib/python3.6/site-packages, outside environment /home/ubuntu/cephtest/ragweed/virtualenv 2023-09-22T13:25:19.625 INFO:teuthology.orchestra.run.smithi096.stdout: Can't uninstall 'pyparsing'. No files were found to uninstall. 2023-09-22T13:25:19.834 INFO:teuthology.orchestra.run.smithi096.stdout: Attempting uninstall: PyYAML 2023-09-22T13:25:19.835 INFO:teuthology.orchestra.run.smithi096.stdout: Found existing installation: PyYAML 3.12 2023-09-22T13:25:19.835 INFO:teuthology.orchestra.run.smithi096.stdout: Not uninstalling pyyaml at /usr/lib64/python3.6/site-packages, outside environment /home/ubuntu/cephtest/ragweed/virtualenv 2023-09-22T13:25:19.835 INFO:teuthology.orchestra.run.smithi096.stdout: Can't uninstall 'PyYAML'. No files were found to uninstall. 2023-09-22T13:25:19.944 INFO:teuthology.orchestra.run.smithi096.stdout: Attempting uninstall: lxml 2023-09-22T13:25:19.944 INFO:teuthology.orchestra.run.smithi096.stdout: Found existing installation: lxml 4.2.3 2023-09-22T13:25:19.944 INFO:teuthology.orchestra.run.smithi096.stdout: Not uninstalling lxml at /usr/lib64/python3.6/site-packages, outside environment /home/ubuntu/cephtest/ragweed/virtualenv 2023-09-22T13:25:19.945 INFO:teuthology.orchestra.run.smithi096.stdout: Can't uninstall 'lxml'. No files were found to uninstall. 2023-09-22T13:25:21.099 INFO:teuthology.orchestra.run.smithi096.stdout:Successfully installed PyYAML-6.0.1 boto-2.49.0 gevent-22.10.2 greenlet-2.0.2 httplib2-0.22.0 importlib-metadata-4.8.3 lxml-4.9.3 munch-4.0.0 pyparsing-3.1.1 typing-extensions-4.1.1 zipp-3.6.0 zope.event-4.6 zope.interface-5.5.2 2023-09-22T13:25:21.192 INFO:teuthology.orchestra.run.smithi096.stderr:+ ./virtualenv/bin/python setup.py develop 2023-09-22T13:25:21.428 INFO:teuthology.orchestra.run.smithi096.stdout:running develop 2023-09-22T13:25:21.466 INFO:teuthology.orchestra.run.smithi096.stdout:running egg_info 2023-09-22T13:25:21.466 INFO:teuthology.orchestra.run.smithi096.stdout:creating ragweed.egg-info 2023-09-22T13:25:21.466 INFO:teuthology.orchestra.run.smithi096.stdout:writing ragweed.egg-info/PKG-INFO 2023-09-22T13:25:21.466 INFO:teuthology.orchestra.run.smithi096.stdout:writing dependency_links to ragweed.egg-info/dependency_links.txt 2023-09-22T13:25:21.466 INFO:teuthology.orchestra.run.smithi096.stdout:writing requirements to ragweed.egg-info/requires.txt 2023-09-22T13:25:21.466 INFO:teuthology.orchestra.run.smithi096.stdout:writing top-level names to ragweed.egg-info/top_level.txt 2023-09-22T13:25:21.466 INFO:teuthology.orchestra.run.smithi096.stdout:writing manifest file 'ragweed.egg-info/SOURCES.txt' 2023-09-22T13:25:21.467 INFO:teuthology.orchestra.run.smithi096.stdout:reading manifest file 'ragweed.egg-info/SOURCES.txt' 2023-09-22T13:25:21.467 INFO:teuthology.orchestra.run.smithi096.stdout:adding license file 'LICENSE' 2023-09-22T13:25:21.468 INFO:teuthology.orchestra.run.smithi096.stdout:writing manifest file 'ragweed.egg-info/SOURCES.txt' 2023-09-22T13:25:21.469 INFO:teuthology.orchestra.run.smithi096.stdout:running build_ext 2023-09-22T13:25:21.469 INFO:teuthology.orchestra.run.smithi096.stdout:Creating /home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages/ragweed.egg-link (link to .) 2023-09-22T13:25:21.469 INFO:teuthology.orchestra.run.smithi096.stdout:Adding ragweed 0.0.1 to easy-install.pth file 2023-09-22T13:25:21.470 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:25:21.470 INFO:teuthology.orchestra.run.smithi096.stdout:Installed /home/ubuntu/cephtest/ragweed 2023-09-22T13:25:21.470 INFO:teuthology.orchestra.run.smithi096.stdout:Processing dependencies for ragweed==0.0.1 2023-09-22T13:25:21.473 INFO:teuthology.orchestra.run.smithi096.stdout:Searching for isodate>=0.4.4 2023-09-22T13:25:21.473 INFO:teuthology.orchestra.run.smithi096.stdout:Reading https://pypi.org/simple/isodate/ 2023-09-22T13:25:21.578 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading https://files.pythonhosted.org/packages/b6/85/7882d311924cbcfc70b1890780763e36ff0b140c7e51c110fc59a532f087/isodate-0.6.1-py2.py3-none-any.whl#sha256=0751eece944162659049d35f4f549ed815792b38793f07cf73381c1c87cbed96 2023-09-22T13:25:21.687 INFO:teuthology.orchestra.run.smithi096.stdout:Best match: isodate 0.6.1 2023-09-22T13:25:21.694 INFO:teuthology.orchestra.run.smithi096.stdout:Processing isodate-0.6.1-py2.py3-none-any.whl 2023-09-22T13:25:21.695 INFO:teuthology.orchestra.run.smithi096.stdout:Installing isodate-0.6.1-py2.py3-none-any.whl to /home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages 2023-09-22T13:25:21.702 INFO:teuthology.orchestra.run.smithi096.stdout:Adding isodate 0.6.1 to easy-install.pth file 2023-09-22T13:25:21.703 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:25:21.703 INFO:teuthology.orchestra.run.smithi096.stdout:Installed /home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages/isodate-0.6.1-py3.6.egg 2023-09-22T13:25:21.808 INFO:teuthology.orchestra.run.smithi096.stdout:Searching for gevent==22.10.2 2023-09-22T13:25:21.808 INFO:teuthology.orchestra.run.smithi096.stdout:Best match: gevent 22.10.2 2023-09-22T13:25:21.808 INFO:teuthology.orchestra.run.smithi096.stdout:Adding gevent 22.10.2 to easy-install.pth file 2023-09-22T13:25:21.808 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:25:21.809 INFO:teuthology.orchestra.run.smithi096.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages 2023-09-22T13:25:21.810 INFO:teuthology.orchestra.run.smithi096.stdout:Searching for munch==4.0.0 2023-09-22T13:25:21.810 INFO:teuthology.orchestra.run.smithi096.stdout:Best match: munch 4.0.0 2023-09-22T13:25:21.810 INFO:teuthology.orchestra.run.smithi096.stdout:Adding munch 4.0.0 to easy-install.pth file 2023-09-22T13:25:21.810 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:25:21.810 INFO:teuthology.orchestra.run.smithi096.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages 2023-09-22T13:25:21.810 INFO:teuthology.orchestra.run.smithi096.stdout:Searching for PyYAML==6.0.1 2023-09-22T13:25:21.810 INFO:teuthology.orchestra.run.smithi096.stdout:Best match: PyYAML 6.0.1 2023-09-22T13:25:21.811 INFO:teuthology.orchestra.run.smithi096.stdout:Adding PyYAML 6.0.1 to easy-install.pth file 2023-09-22T13:25:21.811 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:25:21.811 INFO:teuthology.orchestra.run.smithi096.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages 2023-09-22T13:25:21.812 INFO:teuthology.orchestra.run.smithi096.stdout:Searching for boto==2.49.0 2023-09-22T13:25:21.812 INFO:teuthology.orchestra.run.smithi096.stdout:Best match: boto 2.49.0 2023-09-22T13:25:21.812 INFO:teuthology.orchestra.run.smithi096.stdout:Adding boto 2.49.0 to easy-install.pth file 2023-09-22T13:25:21.812 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:25:21.812 INFO:teuthology.orchestra.run.smithi096.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages 2023-09-22T13:25:21.812 INFO:teuthology.orchestra.run.smithi096.stdout:Searching for six==1.11.0 2023-09-22T13:25:21.812 INFO:teuthology.orchestra.run.smithi096.stdout:Best match: six 1.11.0 2023-09-22T13:25:21.812 INFO:teuthology.orchestra.run.smithi096.stdout:Adding six 1.11.0 to easy-install.pth file 2023-09-22T13:25:21.812 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:25:21.812 INFO:teuthology.orchestra.run.smithi096.stdout:Using /usr/lib/python3.6/site-packages 2023-09-22T13:25:21.813 INFO:teuthology.orchestra.run.smithi096.stdout:Searching for zope.interface==5.5.2 2023-09-22T13:25:21.813 INFO:teuthology.orchestra.run.smithi096.stdout:Best match: zope.interface 5.5.2 2023-09-22T13:25:21.813 INFO:teuthology.orchestra.run.smithi096.stdout:Adding zope.interface 5.5.2 to easy-install.pth file 2023-09-22T13:25:21.814 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:25:21.814 INFO:teuthology.orchestra.run.smithi096.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages 2023-09-22T13:25:21.814 INFO:teuthology.orchestra.run.smithi096.stdout:Searching for setuptools==59.6.0 2023-09-22T13:25:21.814 INFO:teuthology.orchestra.run.smithi096.stdout:Best match: setuptools 59.6.0 2023-09-22T13:25:21.814 INFO:teuthology.orchestra.run.smithi096.stdout:Adding setuptools 59.6.0 to easy-install.pth file 2023-09-22T13:25:21.814 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:25:21.814 INFO:teuthology.orchestra.run.smithi096.stdout:Using /usr/local/lib/python3.6/site-packages 2023-09-22T13:25:21.815 INFO:teuthology.orchestra.run.smithi096.stdout:Searching for greenlet==2.0.2 2023-09-22T13:25:21.815 INFO:teuthology.orchestra.run.smithi096.stdout:Best match: greenlet 2.0.2 2023-09-22T13:25:21.815 INFO:teuthology.orchestra.run.smithi096.stdout:Adding greenlet 2.0.2 to easy-install.pth file 2023-09-22T13:25:21.815 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:25:21.815 INFO:teuthology.orchestra.run.smithi096.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages 2023-09-22T13:25:21.818 INFO:teuthology.orchestra.run.smithi096.stdout:Searching for zope.event==4.6 2023-09-22T13:25:21.818 INFO:teuthology.orchestra.run.smithi096.stdout:Best match: zope.event 4.6 2023-09-22T13:25:21.819 INFO:teuthology.orchestra.run.smithi096.stdout:Adding zope.event 4.6 to easy-install.pth file 2023-09-22T13:25:21.819 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:25:21.819 INFO:teuthology.orchestra.run.smithi096.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages 2023-09-22T13:25:21.819 INFO:teuthology.orchestra.run.smithi096.stdout:Searching for importlib-metadata==4.8.3 2023-09-22T13:25:21.819 INFO:teuthology.orchestra.run.smithi096.stdout:Best match: importlib-metadata 4.8.3 2023-09-22T13:25:21.819 INFO:teuthology.orchestra.run.smithi096.stdout:Adding importlib-metadata 4.8.3 to easy-install.pth file 2023-09-22T13:25:21.819 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:25:21.819 INFO:teuthology.orchestra.run.smithi096.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages 2023-09-22T13:25:21.820 INFO:teuthology.orchestra.run.smithi096.stdout:Searching for zipp==3.6.0 2023-09-22T13:25:21.820 INFO:teuthology.orchestra.run.smithi096.stdout:Best match: zipp 3.6.0 2023-09-22T13:25:21.820 INFO:teuthology.orchestra.run.smithi096.stdout:Adding zipp 3.6.0 to easy-install.pth file 2023-09-22T13:25:21.820 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:25:21.820 INFO:teuthology.orchestra.run.smithi096.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages 2023-09-22T13:25:21.820 INFO:teuthology.orchestra.run.smithi096.stdout:Searching for typing-extensions==4.1.1 2023-09-22T13:25:21.820 INFO:teuthology.orchestra.run.smithi096.stdout:Best match: typing-extensions 4.1.1 2023-09-22T13:25:21.820 INFO:teuthology.orchestra.run.smithi096.stdout:Adding typing-extensions 4.1.1 to easy-install.pth file 2023-09-22T13:25:21.820 INFO:teuthology.orchestra.run.smithi096.stdout: 2023-09-22T13:25:21.820 INFO:teuthology.orchestra.run.smithi096.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages 2023-09-22T13:25:21.820 INFO:teuthology.orchestra.run.smithi096.stdout:Finished processing dependencies for ragweed==0.0.1 2023-09-22T13:25:21.821 INFO:teuthology.orchestra.run.smithi096.stderr:/home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages/setuptools/command/easy_install.py:159: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools. 2023-09-22T13:25:21.821 INFO:teuthology.orchestra.run.smithi096.stderr: EasyInstallDeprecationWarning, 2023-09-22T13:25:21.821 INFO:teuthology.orchestra.run.smithi096.stderr:/home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. 2023-09-22T13:25:21.821 INFO:teuthology.orchestra.run.smithi096.stderr: setuptools.SetuptoolsDeprecationWarning, 2023-09-22T13:25:21.821 INFO:teuthology.orchestra.run.smithi096.stderr:/home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages/pkg_resources/__init__.py:119: PkgResourcesDeprecationWarning: is an invalid version and will not be supported in a future release 2023-09-22T13:25:21.821 INFO:teuthology.orchestra.run.smithi096.stderr: PkgResourcesDeprecationWarning, 2023-09-22T13:25:21.840 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:25:21.841 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/archive/ragweed.client.0.conf 2023-09-22T13:25:21.899 INFO:tasks.ragweed:Configuring boto... 2023-09-22T13:25:21.899 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2023-09-22T13:25:21.899 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/boto.cfg 2023-09-22T13:25:21.957 DEBUG:teuthology.orchestra.run.smithi096:ragweed tests against rgw> RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw' 2023-09-22T13:25:24.467 INFO:teuthology.orchestra.run.smithi096.stderr:ragweed.tests.tests.r_test_multipart_defer_complete.test ... ok 2023-09-22T13:25:25.062 INFO:teuthology.orchestra.run.smithi096.stderr:ragweed.tests.tests.r_test_multipart_defer_update_complete.test ... ok 2023-09-22T13:25:25.518 INFO:teuthology.orchestra.run.smithi096.stderr:ragweed.tests.tests.r_test_multipart_index_versioning.test ... ERROR 2023-09-22T13:25:26.031 INFO:teuthology.orchestra.run.smithi096.stderr:ragweed.tests.tests.r_test_multipart_simple.test ... ok 2023-09-22T13:25:30.636 INFO:teuthology.orchestra.run.smithi096.stderr:ragweed.tests.tests.r_test_obj_storage_class.test ... ok 2023-09-22T13:25:31.734 INFO:teuthology.orchestra.run.smithi096.stderr:ragweed.tests.tests.r_test_obj_storage_class_copy.test ... ok 2023-09-22T13:25:32.950 INFO:teuthology.orchestra.run.smithi096.stderr:ragweed.tests.tests.r_test_obj_storage_class_multipart.test ... ok 2023-09-22T13:25:35.099 INFO:teuthology.orchestra.run.smithi096.stderr:ragweed.tests.tests.r_test_obj_storage_class_multipart_copy.test ... ok 2023-09-22T13:25:35.540 INFO:teuthology.orchestra.run.smithi096.stderr:ragweed.tests.tests.r_test_small_obj_data.test ... ok 2023-09-22T13:25:35.540 INFO:teuthology.orchestra.run.smithi096.stderr: 2023-09-22T13:25:35.541 INFO:teuthology.orchestra.run.smithi096.stderr:====================================================================== 2023-09-22T13:25:35.541 INFO:teuthology.orchestra.run.smithi096.stderr:ERROR: ragweed.tests.tests.r_test_multipart_index_versioning.test 2023-09-22T13:25:35.541 INFO:teuthology.orchestra.run.smithi096.stderr:---------------------------------------------------------------------- 2023-09-22T13:25:35.541 INFO:teuthology.orchestra.run.smithi096.stderr:Traceback (most recent call last): 2023-09-22T13:25:35.541 INFO:teuthology.orchestra.run.smithi096.stderr: File "/home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2023-09-22T13:25:35.541 INFO:teuthology.orchestra.run.smithi096.stderr: self.test(*self.arg) 2023-09-22T13:25:35.541 INFO:teuthology.orchestra.run.smithi096.stderr: File "/home/ubuntu/cephtest/ragweed/ragweed/framework.py", line 345, in test 2023-09-22T13:25:35.541 INFO:teuthology.orchestra.run.smithi096.stderr: self.check() 2023-09-22T13:25:35.541 INFO:teuthology.orchestra.run.smithi096.stderr: File "/home/ubuntu/cephtest/ragweed/ragweed/tests/tests.py", line 296, in check 2023-09-22T13:25:35.541 INFO:teuthology.orchestra.run.smithi096.stderr: index_check_result = rgwa().check_bucket_index(rb.name) 2023-09-22T13:25:35.541 INFO:teuthology.orchestra.run.smithi096.stderr: File "/home/ubuntu/cephtest/ragweed/ragweed/framework.py", line 68, in check_bucket_index 2023-09-22T13:25:35.541 INFO:teuthology.orchestra.run.smithi096.stderr: return self.get_resource('/admin/bucket',{'index' : None, 'bucket':bucket_name}) 2023-09-22T13:25:35.542 INFO:teuthology.orchestra.run.smithi096.stderr: File "/home/ubuntu/cephtest/ragweed/ragweed/framework.py", line 50, in get_resource 2023-09-22T13:25:35.542 INFO:teuthology.orchestra.run.smithi096.stderr: return munch.munchify(json.loads(r.read())) 2023-09-22T13:25:35.542 INFO:teuthology.orchestra.run.smithi096.stderr: File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads 2023-09-22T13:25:35.542 INFO:teuthology.orchestra.run.smithi096.stderr: return _default_decoder.decode(s) 2023-09-22T13:25:35.542 INFO:teuthology.orchestra.run.smithi096.stderr: File "/usr/lib64/python3.6/json/decoder.py", line 342, in decode 2023-09-22T13:25:35.542 INFO:teuthology.orchestra.run.smithi096.stderr: raise JSONDecodeError("Extra data", s, end) 2023-09-22T13:25:35.542 INFO:teuthology.orchestra.run.smithi096.stderr:json.decoder.JSONDecodeError: Extra data: line 1 column 3 (char 2) 2023-09-22T13:25:35.542 INFO:teuthology.orchestra.run.smithi096.stderr:-------------------- >> begin captured stdout << --------------------- 2023-09-22T13:25:35.542 INFO:teuthology.orchestra.run.smithi096.stderr:Munch({'key': 'bucket:test-client.0-r-test-multipart-index-versioning.1', 'ver': Munch({'tag': '_nMoBh2_WUiuqSxbLJ5mw3Jp', 'ver': 1}), 'mtime': '2023-09-22T13:25:25.072052Z', 'data': Munch({'bucket': Munch({'name': 'test-client.0-r-test-multipart-index-versioning.1', 'marker': '42123edd-4a97-43ec-80bc-fc5350f88c11.4533.5', 'bucket_id': '42123edd-4a97-43ec-80bc-fc5350f88c11.4533.5', 'tenant': '', 'explicit_placement': Munch({'data_pool': '', 'data_extra_pool': '', 'index_pool': ''})}), 'owner': 'ragweed.client.0', 'creation_time': '2023-09-22T13:25:25.065311Z', 'linked': 'true', 'has_bucket_info': 'false'})}) 2023-09-22T13:25:35.542 INFO:teuthology.orchestra.run.smithi096.stderr:bucket_info: Munch({'bucket': Munch({'name': 'test-client.0-r-test-multipart-index-versioning.1', 'marker': '42123edd-4a97-43ec-80bc-fc5350f88c11.4533.5', 'bucket_id': '42123edd-4a97-43ec-80bc-fc5350f88c11.4533.5', 'tenant': '', 'explicit_placement': Munch({'data_pool': '', 'data_extra_pool': '', 'index_pool': ''})}), 'creation_time': '2023-09-22T13:25:25.065311Z', 'owner': 'ragweed.client.0', 'flags': 0, 'zonegroup': '570f96c8-d368-4dce-92ba-9b1ab2b076d7', 'placement_rule': 'default-placement', 'has_instance_obj': 'false', 'quota': Munch({'enabled': False, 'check_on_raw': False, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}), 'num_shards': 11, 'bi_shard_hash_type': 0, 'requester_pays': 'false', 'has_website': 'false', 'swift_versioning': 'false', 'swift_ver_location': '', 'index_type': 0, 'mdsearch_config': [], 'reshard_status': 0, 'new_bucket_instance_id': ''}) 2023-09-22T13:25:35.542 INFO:teuthology.orchestra.run.smithi096.stderr:read_test_data= b'{"r_buckets": [{"__pickle": "ccopy_reg\\n_reconstructor\\np0\\n(cragweed.framework\\nRBucket\\np1\\nc__builtin__\\nobject\\np2\\nNtp3\\nRp4\\n(dp5\\nVzone\\np6\\ng0\\n(cragweed.framework\\nRZone\\np7\\ng2\\nNtp8\\nRp9\\n(dp10\\nVconn\\np11\\ng0\\n(cmunch\\nMunch\\np12\\nc__builtin__\\ndict\\np13\\n(dp14\\nVsystem\\np15\\ng0\\n(cragweed.framework\\nRGWConnection\\np16\\ng2\\nNtp17\\nRp18\\n(dp19\\nVhost\\np20\\nVsmithi096.front.sepia.ceph.com\\np21\\nsVport\\np22\\nL80L\\nsVis_secure\\np23\\nI00\\nsg11\\ng0\\n(cboto.s3.connection\\nS3Connection\\np24\\ng2\\nNtp25\\nRp26\\n(dp27\\nVcalling_format\\np28\\ng0\\n(cboto.s3.connection\\nOrdinaryCallingFormat\\np29\\ng2\\nNtp30\\nRp31\\nsVbucket_class\\np32\\ncboto.s3.bucket\\nBucket\\np33\\nsVanon\\np34\\nI00\\nsVsuppress_consec_slashes\\np35\\nI01\\nsVnum_retries\\np36\\nL6L\\nsg23\\nI00\\nsVhttps_validate_certificates\\np37\\nNsVca_certificates_file\\np38\\nV/home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages/boto/cacerts/cacerts.txt\\np39\\nsg22\\nL80L\\nsVproxy\\np40\\nNsVproxy_port\\np41\\nNsVproxy_user\\np42\\nNsVproxy_pass\\np43\\nNsVno_proxy\\np44\\nV\\np45\\nsVuse_proxy\\np46\\nI00\\nsVhttp_exceptions\\np47\\n(chttplib\\nHTTPException\\np48\\ncexceptions\\nOSError\\np49\\ncsocket\\ngaierror\\np50\\nchttplib\\nBadStatusLine\\np51\\ntp52\\nsVhttp_unretryable_exceptions\\np53\\n(lp54\\ncboto.https_connection\\nInvalidCertificateException\\np55\\nasVsocket_exception_values\\np56\\n(L4L\\ntp57\\nsVhttps_connection_factory\\np58\\nNsVprotocol\\np59\\nVhttp\\np60\\nsg20\\ng21\\nsVpath\\np61\\nV/\\np62\\nsVdebug\\np63\\nL0L\\nsVhost_header\\np64\\nNsVhttp_connection_kwargs\\np65\\n(dp66\\nVtimeout\\np67\\nL30L\\nssV_provider_type\\np68\\nVaws\\np69\\nsVprovider\\np70\\ng0\\n(cboto.provider\\nProvider\\np71\\ng2\\nNtp72\\nRp73\\n(dp74\\ng20\\nNsg22\\nNsg64\\nNsV_access_key\\np75\\nVDSXNSOVRUDCFWLDQHZKC\\np76\\nsV_secret_key\\np77\\nVqHdeEmz4nZmu9eBC1PRpmTqjhh6VVUdFaZNSws9DsJT4XZFC9aEf5Q==\\np78\\nsV_security_token\\np79\\nNsVprofile_name\\np80\\nNsVname\\np81\\ng69\\nsVacl_class\\np82\\ncboto.s3.acl\\nPolicy\\np83\\nsVcanned_acls\\np84\\n(lp85\\nVprivate\\np86\\naVpublic-read\\np87\\naVpublic-read-write\\np88\\naVauthenticated-read\\np89\\naVbucket-owner-read\\np90\\naVbucket-owner-full-control\\np91\\naVlog-delivery-write\\np92\\nasV_credential_expiry_time\\np93\\nNsVshared_credentials\\np94\\ng0\\n(cboto.pyami.config\\nConfig\\np95\\ng2\\nNtp96\\nRp97\\n(dp98\\nV_parser\\np99\\ng0\\n(cConfigParser\\nConfigParser\\np100\\ng2\\nNtp101\\nRp102\\n(dp103\\nV_dict\\np104\\nccollections\\nOrderedDict\\np105\\nsV_sections\\np106\\ng105\\n(tRp107\\nsV_defaults\\np108\\ng105\\n(tRp109\\nVworking_dir\\np110\\nV/mnt/pyami\\np111\\nsVdebug\\np112\\nV0\\np113\\nssV_converters\\np114\\ng0\\n(cConfigParser\\nConverterMapping\\np115\\ng2\\nNtp116\\nRp117\\n(dp118\\ng99\\ng102\\nsV_data\\np119\\n(dp120\\nVboolean\\np121\\nNsVfloat\\np122\\nNsVint\\np123\\nNssbsV_proxies\\np124\\ng105\\n(tRp125\\nVDEFAULT\\np126\\ng0\\n(cConfigParser\\nSectionProxy\\np127\\ng2\\nNtp128\\nRp129\\n(dp130\\ng99\\ng102\\nsV_name\\np131\\ng126\\nsVgetboolean\\np132\\ncfunctools\\npartial\\np133\\n(c__builtin__\\ngetattr\\np134\\n(g129\\nVget\\np135\\ntp136\\nRp137\\ntp138\\nRp139\\n(g137\\n(t(dp140\\nV_impl\\np141\\ng134\\n(g102\\ng132\\ntp142\\nRp143\\nsNtp144\\nbsVgetfloat\\np145\\ng133\\n(g134\\n(g129\\ng135\\ntp146\\nRp147\\ntp148\\nRp149\\n(g147\\n(t(dp150\\ng141\\ng134\\n(g102\\ng145\\ntp151\\nRp152\\nsNtp153\\nbsVgetint\\np154\\ng133\\n(g134\\n(g129\\ng135\\ntp155\\nRp156\\ntp157\\nRp158\\n(g156\\n(t(dp159\\ng141\\ng134\\n(g102\\ng154\\ntp160\\nRp161\\nsNtp162\\nbsbssV_delimiters\\np163\\n(V=\\np164\\nV:\\np165\\ntp166\\nsV_optcre\\np167\\ncre\\n_compile\\np168\\n(V\\\\u000a (?P